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Preface 


The emergence of longitudinal data systems over the past decade has provided exciting oppor- 
tunities to better understand students’ educational trajectories over time and institutions. Be- 
ginning with the National Student Clearinghouse data, researchers have been able to follow 
large numbers of students from high school into postsecondary institutions. This research, most 
notably the studies conducted by the Consortium for Chicago School Research, has produced 
important lessons about what matters most in the way high schools prepare their students for 
their educational futures.! 


At the same time, with states like Florida and Texas pointing the way, statewide longitudinal data 
systems have rapidly developed and expanded. Almost every state now has the capacity to chart the 
trajectories of its students across time and school systems. States like Indiana and Kentucky are pro- 
ducing valuable feedback reports that inform high schools how well they have prepared students for 
postsecondary success. 


Policy analysts also are using these sources of data—and other longitudinal data sets—to inform 
policies designed to stimulate and support districts and schools in their efforts to prepare young 
people for postsecondary success. An analysis of the relationship between postsecondary perfor- 
mance and adequate yearly progress found that looking at how students did in their postsecond- 
ary studies beyond high school—whether they enrolled, whether they needed remedial course 
work, what grades they earned and whether they returned for a second year—provided a more 
accurate measure of a school’s ability to prepare its students than performance on standardized 
tests.” As the author points out, incorporating these new data points has the potential to cre- 
ate more powerful accountability systems for secondary education and increases the pressure 
on schools that do a poor job of preparing their students for postsecondary success. Florida 
already has incorporated postsecondary indicators in its accountability structure for assessing 


high schools’ progress. 


Still, a Middle Eastern proverb offers wise caution that “you can’t fatten a cow by weighing it.” As 
helpful as all this new data capacity can be, the numbers—and the students they represent—will 
not change if the data are not used by those who are in a position to shape students’ lives. These 
are the building-level staff—teachers, counselors, and administrators—who need to understand 
what the research teaches us about the factors that are likely to lead to postsecondary readiness, 
access, and success. They then need to build on that understanding by examining data about 
their own students’ performance in high school and beyond, looking for patterns that can help 
point the way to needed changes in practice. This is not a once-a-year exercise, nor an analysis 
done by the principal and then shared with faculty at a staff meeting. Good data practice—the 
kind that leads to changes in the classroom and eventually changes in student outcomes—re- 
quires time and collaboration. 


A regular examination of collective or aggregate analysis (e.g., the number of students 
at or above standard) not only promotes a common goal orientation but also brings 
forth the insights of many minds. Such analysis breaks down the cellular structure of 


1 


See, for example, Melissa Roderick et al., From High School to the Future: A First Look at Chicago Public School 
Graduates’ College Enrollment, College Preparation, and Graduation from Four-Year Colleges (Chicago: Consortium 
for Chicago School Research, 2006) and Melissa Roderick et al., From High School to the Future: Potholes on the 
Road to College (Chicago: Consortium on Chicago School Research, 2008). 


* Chad Aldeman, College- and Career-Ready: Using Outcomes Data to Hold High Schools Accountable for Student 
Success (Washington, DC: Education Sector, 2010). 
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schools and brings forth a precious perspective that can only be heard in communion 
with others whose struggles are similar.’ 


This guide has been written with this level of practice in mind. 


Acknowledgements 


Many people played important roles in the development of this guide. The authors wish to thank 
Wendy Douglas-Nathai for everything from tracking references and preparing tables to organizing 
the logistics of site visits. We also wish to thank the full technical assistance team working to support 
the US. Department of Education’s Smaller Learning Communities program, a collaboration of 
four organizations (The Millennium Group International, LLC, Great Schools Partnership, the 
Center for Secondary School Redesign, and FHI 360), which provided thoughtful feedback to an 
earlier draft of this guide. Most of all, we wish to thank Angela Hernandez-Marshall and Braden 
Goetz from the US. Department of Education, whose guidance and encouragement have been 
invaluable in shepherding this work. 


3 Mike Schmoker, Results: The Key to Continuous Improvement (Alexandria, VA: ASCD, 1999). 


From Information to Action: A Guide to Using Postsecondary Data iii 


Introduction 


The stark reality for today’s high school students is that some form of postsecondary education is a 
necessary ticket of entry to a secure economic future. Figure 1, below, shows the educational back- 
ground of workers with incomes in the middle four family-income deciles ($30,000 to $79,000 in 
2007). More than 60 percent had at least some college education, while fewer than four in 10 of those 
in this income range had a high school diploma or less. 


Figure 1. Educational composition of the middle class workforce, 2007 
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SOURCE: Anthony P. Carnevale, Nicole Smith, and Jeff Strohl, Help Wanted: Projections of Jobs and Education Re- 
quirements Through 2018 (Georgetown Center on Education and the Workforce, 2010). 


This pattern of increasing education requirements for well-paid employment is expected to continue. 
According to the Georgetown University Center on Education and the Workforce: 


By 2018, our forecasts show the economy will create 46.8 million openings—13.8 brand new jobs 
and 33 million replacement jobs, positions vacated by workers who have retired. Nearly two-thirds of 
these 46.8 million jobs—some 63 percent—will require workers with at least some college education. 
About 34 percent will require a Bachelor's degree or better, while 30 percent will require at least some 
college or a two-year Associate’s degree.* 


Even following the 2007-2009 recession, recent graduates are doing much better economically than 
their peers who did not pursue a postsecondary education. In 2010, 88 percent of four-year college 
graduates were employed, earning an average weekly wage of $581. For those with only a high school 
diploma, only 64 percent were employed, with an average weekly wage of $305.” 


Over a lifetime, students who complete postsecondary education and training will earn far more than 
their less-educated peers. Figure 2 shows the ratio of lifetime earnings of each degree in comparison to 


* Anthony P. Carnevale, Nicole Smith, and Jeff Strohl, Help Wanted: Projections of Jobs and Education Requirements 
Through 2018 (Washington, DC: Georgetown Center on Education and the Workforce, 2010), 110. 


> Michael Greenstone and Adam Looney, “How Do Recent College Grads Really Stack Up? Employment and Earnings 
for Graduates of the Great Recession,’ Brookings on Job Numbers Blog, June 3, 2011, http://www.brookings.edu/ 
opinions/2011/0603_jobs_greenstone_looney.aspx. 


From Information to Action: A Guide to Using Postsecondary Data 1 


a high school diploma. For example, the lifetime earnings of someone with a master’s degree are likely 
to be nearly twice the earnings of a high school graduate. 


Figure 2. Expected lifetime earnings relative to high school graduation, by education level 
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SOURCE: National College Access Network, Creating and Operating a Statewide, Regional or Citywide College Access 
and Success Network, 2011, p. 6. 


Increasing the proportion of the population with postsecondary education also is critical to the na- 
tion’s economic future. Based on economists’ predictions, the Lumina Foundation warns: 


What is now very clear is that, when structural job loss takes place in an economy with increasing skill 
requirements — such as ours, education and training are essential to putting people back to work. If 
we can’t supply labor markets with enough people who have the necessary knowledge and skills, eco- 
nomic growth will be choked off: 


Using Data as a Tool to Increase Postsecondary Success 


Given the extent to which “the labor market clearly has linked middle-class employability to postsecondary 
education and training,” educators around the country are working to increase students’ readiness for, ac- 
cess to, and success at the postsecondary level. Concurrent with the push for higher levels of postsecondary 
education, the last decade has brought greater availability and comparability of data on student perfor- 
mance and of school and district use of data to inform efforts to improve student achievement. 


6 


Lumina Foundation, A Stronger Nation Through Higher Education, March 2012, 5. 
7 Carnevale, Smith, and Strohl, Help Wanted, 110. 
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Two important sources of data on postsecondary success are the state-level longitudinal databases 
(SLDS) and the National Student Clearinghouse (NSC), which launched its secondary research ini- 
tiative in 2009. These systems combine information from multiple sources for each individual stu- 
dent, allowing performance to be studied across time, schools, and institutional levels. In 2005 and 
2009, the federal government invested in developing and expanding SLDS across the country. A pri- 
mary goal of the investment was to provide researchers, policy makers, and practitioners the capacity 
to examine how well high school graduates are doing at the next level. High schools will be able to 
learn where their students enroll after high school and what they study, whether or not they need 
additional course work before taking credit-bearing classes, how well they do, how long they persist 
in their education and training efforts, whether or not they transfer to other institutions, and what 
degrees, certificates, or other forms of credentials they earn. 


Even though more data are available, too many high schools have little reliable knowledge about what 
happens to their graduates. Most survey their seniors about their future plans, but this is self-reported 
data, and before the fact. (Unfortunately, the new sources of data have shown that too many students, 
particularly first-generation college-goers, do not enroll in postsecondary education even when they 
have been accepted.*) Nor do many high schools learn how well those students who enroll in college 
do in their course work or whether they need to take remedial classes before they can begin accumu- 
lating credits toward a degree or certificate. 


How Can Postsecondary Data Be Useful? 


Using postsecondary data can help educators better understand important patterns of student 
achievement and difficulty, determine priorities for instructional improvement, and assess whether 
their implemented improvements have made a difference. By zeroing in on which groups of students 
do or do not succeed at the postsecondary level, educators can identify how to strengthen high school 
preparation for postsecondary education. Here are just a few examples: 


e Data can reveal unknown problems that block student success. A 2008 study of postsecond- 
ary transitions by the Consortium for Chicago School Research found that roughly one in 
five students who had been accepted to a four-year college did not actually enroll in college.’ 


e Data can help pinpoint barriers to postsecondary access. The same study by the Consortium 
for Chicago School Research found that students who completed their Free Application for 
Federal Student Aid (FAFSA), which is designed to determine students’ eligibility for federal 


financial aid, were twice as likely to enroll in a four-year college as those who did not. 


e Data can help schools see the need for deeper change. In Tennessee, the Hamilton County 
Department of Education and the Chattanooga Public Education Foundation examined data 
from the two postsecondary institutions that received the greatest share of high school gradu- 
ates, learning that half of these students had to enroll in remedial mathematics classes. This, 
in turn, led to an intensive strategy for improving secondary-level mathematics instruction. 


e Data can overturn incorrect assumptions. Two recent longitudinal analyses of the qualifica- 
tions of students enrolling in postsecondary education have challenged the assumption that 
students enroll in the most selective institutions to which they are accepted, showing that 


8 


Harvard Strategic Data Project, The College Match: Do High School Graduates Enroll in Colleges that Maximize 
Their Chances of Success? April, 2012. 


? Roderick, From High School to the Future, 2008, 37. 
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roughly one-third enrolled in institutions for which they were overqualified.’° Both studies 
also found that these students were significantly less likely to graduate than academically simi- 
lar students who attended institutions that matched their qualifications. 


These findings and those like them have informed the efforts of school leaders, teachers, and guid- 
ance and advisory personnel to help students prepare for postsecondary education and negotiate 
the process of identifying postsecondary options, applying for admission, and securing financial 
assistance. Understanding the intervention points on the pathway from high school to postsecond- 
ary enrollment can help increase the proportion of students who not only aspire to a postsecondary 
education—as so many of today’s young people already do—but also take the right steps to make 
see their aspirations realized. 


What Kind of Data Analysis Is Involved? 


The current and growing abundance of data from multiple sources is daunting, particularly to those 
at the school level. Where should school staff begin? What data should they use? What level of 
analysis will be needed to yield effective guidance for positive change? As a way to consider the 
varying complexity of data, a 2009 guide to collecting and using postsecondary data proposed three 
levels of analytic work: highly detailed data analysis, detailed data analysis, and basic data analysis."' 


The most complex, or what the guide refers to as “highly detailed data analysis,” requires the ability 
to manipulate and match multiple data sources and to conduct very sophisticated statistical analy- 
ses, using techniques such as hierarchical linear modeling. These analyses usually are conducted by 
researchers with access to large datasets and significant resources to support their investigations. 


The guide calls the intermediate level “detailed data analysis.” These analyses combine basic stu- 
dent data and postsecondary data with transcript data and employ analytic approaches that involve 
“statistical tests (t-tests or nonparametric tests such as chi-square) to examine the relationship of 
one variable (such as the number of math courses completed in high school) with another (college 
enrollment).” These analyses could be done by district-level data analysts with access to the data and 
knowledge about which statistical tests are appropriate. 


The third level is “basic data analysis.” While less complex from an analytic perspective, this level 
includes high leverage work at the school level. Administrators, teachers, and counselors review 
data that can easily be collected from their local district and from state or national postsecond- 
ary sources. They also can supplement these data with surveys and focus groups. These are “basic 
descriptive analyses (counting how many students do something numerically or by averages or per- 
centages).” They can be completed using a spreadsheet program and displayed in easily interpreted 
tables and charts. 


These different levels of research are interrelated. Very detailed analyses yield important research 
findings that guide the work of both practitioners and policy makers. Detailed data analyses can 
be used to evaluate the effectiveness of interventions that attempt to incorporate these findings. 
Both kinds of analyses help practitioners understand which elements are most likely to help them 
achieve desired outcomes. 


Practitioners and district-level research personnel cannot and need not conduct highly detailed data 


0 William G.W. Bowen, Matthew M. Chigos, and Michael S. McPherson, Crossing the Finish Line: Completing College 
at America’s Public Universities (Princeton, NJ: Princeton University Press, 2009). 


Neal Finkelstein et al., Improving the Quality and Use of Postsecondary Data: A Guide for Smaller Learning Com- 
munity Grantees (Portland, OR: Northwest Regional Educational Laboratory, U.S. Department of Education, 2009), 
28-31. 
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analyses such as those done by the Consortium for Chicago School Research. They have neither the 
time nor the resources. The great value of these analyses is that they lead the field to better understand 
what factors others can monitor using less complex analytic approaches. When district research staff 
and school personnel are informed research consumers, they are more likely to apply research findings 
from these cutting-edge studies in their own efforts to improve student outcomes. 


What Level of Technical Quality Do Postsecondary Data Need for Different Purposes? 


A common concern among education practitioners is whether the data they are receiving are 
reliable and valid. This is an important concern, particularly for two purposes. For researchers 
who are trying to establish statistically significant relationships between different data elements, 
completeness and accuracy of the data are essential. Data accuracy also matters for teachers who 
are trying to tailor instructional strategies for individual students. 


This guide, however, focuses on the use of data to reveal patterns of achievement and difficulty 
and the differential distribution of access and obstacles to postsecondary education. These pat- 
terns and distributions can provide guidance to policy makers and practitioners about where to 
focus their efforts across larger groups of students. While it is preferable to have clean and ac- 
curate data to the greatest extent possible, analyses of these sorts can survive minor problems of 
incompleteness. 


What is important, however, is that the data be timely and able to be disaggregated (by gender, 
race/ethnicity, income, and disability), both by individual and combinations of student charac- 
teristics. This is essential for schools to examine which students are succeeding and which are 


having difficulty. 


What Do We Know about Effective Use of Data? 


The growing availability of different kinds of data opens up exciting opportunities for educators. But 
it also presents daunting challenges as they try to make meaning from the data. Studies of effective use 
of data in the past decade provide some important lessons:' 


e Keep data presentation simple. Simple charts and tables that focus on a few important ques- 
tions are more powerful in producing school-level change than complicated data reports that 
only experts can use. 


© Make reviewing data part of regular school discussions. Frequent examination of data is needed 
to develop a culture of inquiry that can produce changes in practice. 


e Discuss the data. Collective discussion of data involving groups of educators leads to broader 
change than detailed analysis by individual teachers. 


e Use multiple forms of data. Data from administrative sources like assessment results and post- 
secondary enrollment and performance data are useful but need to be supplemented by other 
sources of data that help educators better understand the reasons behind the results. These 
sources can include data from teacher-developed assessments, as well as from/ rubrics, sur- 
veys, and focus groups. 


2 Mary Ann Lachat and Stephen Smith, Data Use in Urban High Schools (The Education Alliance at Brown University, 
2004); Amanda Datnow, Vicki Park, and Brianna Kennedy, Acting on Data: How Urban High Schools Use Data to 
Improve Instruction (Los Angeles, CA: Center on Educational Governance, USC Rossier School of Education, 2008). 
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Audience for the Guide 


This guide has been prepared with three main audiences in mind: 


e Secondary school and district-level personnel who want to understand how their graduates are 
doing in order to improve their students’ readiness for, access to, and success in postsecondary 
education and training. 


e Institutions of higher learning, particularly those that are building relationships with their 
feeder secondary schools and that often receive the largest share of any high school’s gradu- 
ates. They may benefit from guidance that helps them learn about the characteristics of their 
incoming students. As an example, in Texas, which has been a national leader in creating 
a longitudinal database that crosses the boundary between K12 and higher education, the 
Austin and Amarillo school districts have built strong data-sharing partnerships with area 
colleges and universities. 


© Community-based organizations, local education foundations, and other organizations that are 
working in partnership with secondary schools to increase postsecondary access and success. 
In Tennessee, for example, a partnership between the Chattanooga Public Education Foun- 
dation and the Hamilton County Department of Education led to a focus on increasing not 
only high school graduation but also the proportion of students entering postsecondary edu- 
cation without the need for remedial course work. 


Layout of the Guide 


We begin the guide with a brief discussion of the emerging conceptualizations of the precursors of 
postsecondary success. While many of the constructs in these frameworks are not yet easily measur- 
able, the discussion forms an important backdrop for work designed to increase postsecondary readi- 
ness, access, and success. 


Chapter Two turns to the ways we now are able to measure these concepts, based on highly detailed 
analyses of large datasets on the postsecondary careers of thousands of students. The findings from 
these studies identify the patterns of interest that can serve to guide school-level analyses of postsec- 
ondary data. 


Chapter Three discusses the primary sources of administrative data that are currently available to 
schools, along with some of the opportunities and challenges associated with each of these sources. 


Chapter Four turns to recent efforts to secure detailed data on students’ postsecondary outcomes 
from local institutions. This chapter includes brief profiles of two such efforts, one of which illustrates 
the tremendous potential of using longitudinal data and the second of which shows what can be done 
even when longitudinal data for individual students are not available. 


The next three chapters discuss practical ways to collect and use data. Chapter Five highlights the 
use of surveys and focus groups to gain additional insight into the findings produced by using longi- 
tudinal data. Chapter Six turns to the process of framing questions for inquiry, identifying the data 
needed to answer them, and making a request for data compliant with legal protection of student 
records. Chapter Seven discusses the process for working with data at the practitioner level. This chap- 
ter concludes with a profile of a consortium of high schools that worked with an outside partner to 
understand longitudinal data findings on the postsecondary outcomes of their graduates and to learn 
data-coaching skills that could be used to lead change in each of the schools. 
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CHAPTER ONE 


Thinking About Postsecondary 


Readiness, Access, and Success 


This chapter discusses leading thinking about what constitutes good preparation for postsecondary 
success. After clarifying what we mean by postsecondary education, we explain why we are looking at 
readiness and access as separate, though related, ideas. The chapter will conclude by briefly describing 
two leading conceptualizations of college and career readiness. 


What Do We Mean by Postsecondary Education? 


Most people think of two-year and four-year colleges as the primary postsecondary options, but post- 
secondary education can and should include a wide range of institutions and desired outcomes. In- 
stitutions of higher education encompass two-year and four-year colleges and graduate institutions, 
both public and private, as well as trade schools and vocational programs in every state in the nation, 
Washington, D.C., and the US. territories. The overwhelming majority of jobs in the future economy 
will require some form of higher education, but not all of them will call for a four-year bachelor’s 
degree or beyond. By 2018, an estimated 29 percent of new jobs will require an associate’s degree or 
professional certificate. 


What Is the Difference between Increasing Readiness and Increasing Access? 


Most discussions of how to increase postsecondary participation among today’s students focus on one 
or both of two arenas for action. The first involves academic preparation for the course work students 
will need to take at the postsecondary level and the cognitive content and skills they will need to mas- 
ter to do well in postsecondary classes. We refer to this as postsecondary readiness. The second involves 
the process of learning about postsecondary options, selecting and applying to the postsecondary 
institutions for which one is qualified, and securing financial assistance, if needed. We refer to this 
as postsecondary access. Some aspects of these two sets of activities are interrelated, but to clarify what 
each one entails, we will talk about these two areas of focus separately. 


How Does Readiness Affect Postsecondary Success? 


There is a difference between indicators of college readiness and evidence of readiness. The former is 
assessed while students are still in high school through the use of both quantitative and qualitative 
measures. The latter comes from data that tell us what really happened after a student leaves high 
school, including college enrollment, matriculation, persistence, and completion.* 


Since 2005, thanks to the availability of school-level analyses from the National Student Clearing- 
house (NSC), which now collects data from 96 percent of the nation’s postsecondary institutions, 
some high schools have been able to collect the evidence of college readiness among their graduates. 
Combined with data at the individual and school levels from a range of sources, the NSC data enable 
school districts and individual schools to ask important questions about how well they are preparing 
students for postsecondary education and training. 


‘5 Carnevale, Smith, and Strohl, Help Wanted, 14 


‘Anne Hyslop, Data That Matters: Giving High Schools Useful Feedback on Grads’ Outcomes (Washington, DC: 
Education Sector, November 2011). 
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“Increasing qualifications is the single most important strategy for improving college access and col- 
lege success.” This conclusion from research by the Consortium for Chicago School Research indi- 
cates that students’ high school preparation is correlated with their success at the postsecondary level. 
However, the same body of research found considerable variation at the postsecondary level in how 
well different institutions educate the same kinds of students. In a 2006 study of the postsecondary 
trajectories of Chicago Public School graduates, institutional graduation rates for students with simi- 
lar levels of high school preparation and economic background varied widely.'* This suggests that we 
need to be cautious in attributing student postsecondary performance beyond the freshman year to 
the quality of high school preparation. However, similar to the importance of ninth grade success for 
high school graduation, how well students do in their first year of postsecondary studies is a strong 
predictor of whether they will earn a degree or a certificate. 


An Evolving Understanding of College and Career Readiness 


Recent longitudinal analyses of data on student performance in postsecondary education have illu- 
minated the pathway to postsecondary success. Beginning in 1999 with Clifford Adelman’s Answers 
in the Tool Box: Academic Intensity, Attendance Patterns, and Bachelors Degree Attainment, research- 
ers have been able to examine how variations in student preparation were associated with differing 
outcomes at the postsecondary level. Adelman’s 2006 study, The Toolbox Revisited, confirmed and 
expanded his original findings. 


At the same time, other scholars and practitioners have been trying to better understand and describe 
what constitutes appropriate preparation for postsecondary education. For the most part, these dis- 
cussions involve knowledge and skills that are more difficult to measure than the elements in Adel- 
man’s research. 


David Conley identifies four components that constitute college and career readiness, as shown in 
figure 3. The first two columns show what Conley believes to be the essential knowledge and skills 
students must have to succeed in their academic work. These include the more traditional academic 
content knowledge and high-level cognitive skills that are needed for most postsecondary education 
course work. The third column includes self-management and study skills that a student will need to 
mobilize to succeed. The fourth column shows a set of knowledge and skills that students need to gain 
access to a postsecondary institution and to manage their academic life in that institution. 


Figure 3. David Conley’s conceptual framework for college and career readiness 
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Roderick, From High School to the Future, 2006, 90. 


a 


Ibid., 80-83. 
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Key Learning Skills and Key Transition 
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SOURCE: David T. Conley, Defining and Measuring College and Career Readiness, Educational Policy Improvement 
Center, University of Oregon. Presented at the Council of Chief State School Officers’ PNxGL Innovation Labs Silver 
Spring, MD, September 21, 2011, Slide 9. 


In Conley’s review and analysis of efforts to define measurable indicators of college and career readi- 
ness, he sets this measurable benchmark for assessing college and career readiness: 


The level of preparation a student needs in order to enroll and succeed—without 
remediation—in a creditbearing course at a postsecondary institution that offers 
a baccalaureate degree or transfer to a baccalaureate program, or in a high-quality 
certificate program that enables students to enter a career pathway with potential 
future advancement. 


He further specifies: 


Success is defined as completing the entry-level courses or core certificate courses at a 
level of understanding and proficiency that makes it possible for the student to con- 
sider taking the next course in the sequence or the next level of course in the subject 
area or of completing the certificate.’” 


Parallel to and drawing upon Conley’s work, the John Gardner Center at Stanford University and 
the Annenberg Institute for School Reform at Brown University have been collaborating on a set of 
college readiness indicator systems or CRIS. For them, “College Readiness consists of three distinct 
yet interdependent dimensions: Academic Preparedness (AP), Academic Tenacity (AT), and College 
Knowledge (CK).” Each of these dimensions has two theoretical constructs: 


Academic Preparedness: 


1) Academic knowledge and skills necessary to succeed in college-level courses 


2) Key cognitive strategies necessary to succeed in college-level courses 


Academic Tenacity: 


3) Behaviors of active participation and perseverance through adversity 


4) Beliefs, attitudes, and values that prioritize success in school and drive student engagement and work 


7 David T. Conley, Defining and Measuring College and Career Readiness (University of Oregon: Educational Policy 
Improvement Center, PowerPoint Presentation, 2011). 
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College Knowledge: 


5) Knowledge, skills, and behaviors apart from academic content that allow students to success- 
fully access college 


6) Knowledge, skills, and behaviors apart from academic content that allow students to succeed 
once in college’* 


Consistent with the Conley and CRIS frameworks, a recent report on noncognitive factors in ado- 
lescent learning by the Consortium for Chicago School Research identified the five most important 
noncognitive elements linked to academic performance: academic behaviors, academic perseverance, 
academic mindsets, learning strategies, and social skills.!? These conceptualizations are helpful in 
shaping and enriching our thinking about how best to prepare high school students for the challenges 
of accessing and succeeding at the postsecondary level. However, we do not have an easily used set of 
indicators for measuring how well schools and students are doing in their efforts to increase college 
and career readiness that align with either of these conceptual frameworks. 


That does not mean, however, that schools trying to implement postsecondary readiness as described above 
are without data to guide their efforts. In addition to the more familiar sources of data (e.g,, assessment 
results, course enrollment, and grades), they can collect data about the extent to which their school has of- 
fered students access to the practices described above that are consistent with these conceptualizations of 
effective postsecondary preparation. This was the case in one of Miami’s high schools, as described below. 


Using Qualitative Data to Guide Practice: 
Westland Hialeah High School 


Westland Hialeah is one of three high schools in Miami working with the Citi Post- 
secondary Success Program (CPSP), a Citi Foundation five-year grant to create a 
college-going culture, particularly among low-income and/or first-generation stu- 
dents. Through the grant, the Miami Education Fund provides support and services 
to Westland Hialeah to increase the college-going rates and decrease the barriers to 
college for many of its students. 


As part of CPSP, Westland Hialeah High School participated in an asset mapping anal- 
ysis using a tool developed by the CPSP technical support partner, FHI 360.” The tool 


is designed to help schools improve the coordination, quality, and reach of good prac- 
tices already in place; to identify gaps; and to develop goals for improving college access 
and success among their students. Westland Hialeah’s asset analysis identified trends 


'§ College Readiness Indicator Systems (CRIS) Kick-Off Convening Materials, Glossary and Reference Document, April 
4-5, 2011. 


Camille A. Farrington et al, Teaching Adolescents To Become Learners: The Role of Noncognitive Factors in Shaping 
School Performance: A Critical Literature Review (Chicago: Consortium for Chicago School Research, 2012). 


*) The asset analysis was conducted using a tool developed by the CPSP technical support partner, FHI 360; it can be found 
at http://www.fhi360.org/resource/navigating-college-readiness-through-asset-mapping-fact-sheet. This tool is based on 
David Conley’s research on college readiness and organized into four categories: academic behavior, cognitive skills, con- 
textual skills and content knowledge. It also draws from Educators for Social Responsibility’s work on increasing college 
access, particularly the development of a continuum of support starting in the ninth grade, as discussed in “A Suggested 
Outline for Postsecondary Preparation; from Increasing College Access through School-Based Models of Postsecondary 
Preparation, Planning, and Support. (Cambridge, MA: ESR, January 2009); as well as the Pathways to College Network 
(PCN) College Readiness for All Toolbox, accessed December 10, 2009, http://toolbox.pathwaystocollege.net/. 
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within the school as well as barriers, opportunities, and gaps in services, support, and 
access. It also pointed to additional data collection and analysis needs. 


Based on the asset analysis work, Westland Hialeah developed a comprehensive ac- 
tion plan for improving student preparation and readiness for college. The action plan 
focused on three areas of college readiness: 1) key content knowledge; 2) contextual 
skills and awareness; and 3) academic behaviors. For each area of college readiness, 
Westland Hialeah identified three specific skills that needed to be developed either in 
students or staff or in the school support and structure. For each skill that Westland 
Hialeah chose to work on, they developed an action plan. 


One of the most significant goals that came out of the first asset mapping activity was 
Westland Hialeah’s decision to begin preparing its students for college in the ninth 
grade. Staff redesigned the goals and content of their ninth grade orientation class to 
put students on the track to postsecondary success from the very beginning. Every 
student at Westland Hialeah takes the Freshman Seminar. This year-long required 
course walks students through the process of setting up their FACTS.org accounts”! 
and guides them through the ePersonal Education Planner (ePEP). The ePEP planner 
helps students monitor their progress through high school and compares their tran- 


script information with current requirements for graduation, scholarships, and even 


state university admissions. Students also develop postsecondary career goals and 
aspirations and explore the Florida postsecondary system in their freshman course. 
They complete a term paper and give a presentation that focuses on their projected 
career choices, using the data and information they have gathered about that field. 


The asset mapping tool also calls attention to students’ opportunities to participate 
in dual enrollment programs and earn college credit offered by local colleges and uni- 
versities. At Westland Hialeah, nearly 12 percent of the school’s 1,950 students were 
taking college course work through Florida International University in 2012. An ad- 
ditional 94 students attended classes at Miami-Dade College. According to the prin- 
cipal, the dual enrollment programs reflect a change in the culture of the school and 
have helped Westland Hialeah improve its grade on the state accountability system 
from a B to an A (Florida’s high school framework gives schools credit for student 
participation and performance in accelerated course work).” 


While we wait for better measures that assess students’ readiness along the lines suggested by Conley 
and CRIS, the most useful currently available measures for looking at postsecondary readiness come 
from the two primary college-eligibility testing services: the ACT and the College Board, which of- 
fers the SAT examination. Both have done research on how well their tests predict postsecondary suc- 
cess, and recently, each has specified a level of performance on their tests that correlates to the ability 
to place into and succeed in credit-bearing classes at the postsecondary level. 


For the SAT, a combined score of 1550 on the Critical Reading, Mathematics, and Writing tests 
“indicates a 65 percent likelihood of achieving a B- average or higher during the first year of college, 


*! FACTS.org is Florida’s Academic Counseling and Tracking for Students. Sponsored by the Department of Education 
and the Florida Center for Advising & Academic Support, this free online advising website helps students plan and 
track their education progress from middle school through college. 


” Laura Isensee, “Getting a Jump on College (at a Discount); Miami Herald, posted Feb. 22, 2012. 
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which in turn is indicative of a high likelihood of college success and completion.”? The ACT research 
set benchmarks that predicted a 50 percent likelihood of achieving a B or higher and a 75 percent 
likelihood of achieving a C in first-year credit-bearing courses with ACT scores of 18 in English, 21 
in Reading, 22 in Mathematics, and 24 in Science. 


The next chapter focuses on the ways we currently are able to assess college readiness, access, and success, 
as well as important questions schools can ask about how their students are doing on these measures. 


College Board, “43% of 2011 College-Bound Seniors Met SAT* College and Career Readiness Benchmark, September 
14, 2011) press release, 2. 


** ACT, The Condition of College and Career Readiness, 2011, http://www.act.org/readiness/2011. 
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CHAPTER TWO 


Key Leverage Points on the Pathway to 
Postsecondary Success 


This chapter discusses elements that can be measured, using multiple sources of data, at critical junc- 
tures along the way to postsecondary success. It also articulates the key questions that need to be asked 
about each juncture as part of efforts to improve postsecondary access and success. 


As the push for higher education and postsecondary training has grown, education policy makers and 
practitioners have worked to identify what schools and communities can do to increase postsecond- 
ary access and success. Figure 4 shows what we now can measure along the pathway to postsecondary 
success, based on available data elements and definitions. 


Figure 4. Key leverage points on the pathway to postsecondary success 


Postsecondary Readiness 
Take the core curriculum 


Maintain a high GPA 
Take higher level classes 
Score well on ACT/SAT 


Postsecondary success 
Take credit-bearing 
classes 
Accumulate 20 credits 
Complete a 2" year 
Complete program 


Postsecondary Access 
Aspire to postsecondary education or 


training (PSE/T) 
Plan to apply to PSE/T 
Complete application 
- Apply to right type of PSE/T 
+ Apply for financial aid (FAFSA) 
Enroll in PSE/T 


Postsecondary Readiness 
Four Leverage Points of Postsecondary Readiness 


Using Adelman’s research on the postsecondary trajectories of thousands of students as well as the 
studies done by both the College Board and ACT, we can identify four elements that constitute a set 
of measures of postsecondary readiness: 


e ‘Take the core curriculum classes that are generally recommended as part of a pre-college cur- 
riculum. 

e ‘Take additional higher-level classes, particularly in mathematics and science. 

e Maintain a grade point average of B or better. 

e Score well on the ACT or SAT tests. 
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Take the core curriculum and additional higher-level classes: In both Adelman studies (1999 and 
2006), the academic curriculum intensity of a student’s preparation was the single most powerful pre- 
dictor of postsecondary success in completing a four-year college degree. (Adelman points out some 
students attend schools where they do not have access to all such courses.) In both studies, students 
who succeeded at the four-year college level had completed at least: 

e 3.75 or more Carnegie units of English 

e 3.75 or more Carnegie units of mathematics 

e Highest mathematics of either calculus, pre-calculus, or trigonometry 


e 2.5 or more Carnegie units of science or more than 2.0 Carnegie units of core laboratory sci- 
ence (biology, chemistry, and physics) 


e More than 2.0 Carnegie units of foreign languages 
e More than 2.0 Carnegie units of history and/or social studies 
e More than 1 Advanced Placement course 
e No remedial English; no remedial mathematics” 
An ACT analysis of the high school preparation of students who took the ACT in 2011 shows how 


important these choices are to meet the ACT Benchmarks for postsecondary success. 


Figure 5. Percentage of ACT-tested high school graduates meeting ACT college readiness bench- 
marks by number of years of courses taken within subjects, 2011 


of - 
80% 7 68% 
60% 
40% ~- 

y a " a 
20% Core 

™ Less than core 
0% 7 y 
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Percent achieving benchmark by subject 


SOURCE: ACT, The Condition of College and Career Readiness, 2011, Slide 6. 


It should be noted that these are the levels of course work associated with success at the four-year col- 
lege level. Research on predictors of success at the two-year college level provides less guidance, in part 
because the range of students who attend these institutions varies so widely in age and preparation. 
However, Adelman’s 2005 analysis of the postsecondary outcomes of community college students 
found that avoiding placement in remedial classes was critical to eventual completion, and that pass- 


> Clifford Adelman, The Toolbox Revisited: Paths to Degree Completion From High School Through College (Wash- 
ington, DC: US. Department of Education, 2006), 27. 
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ing a minimum of Algebra 2 in high school was required to complete at least one college-level math- 
ematics course at the community college level.” 


Maintain a high grade point average: Adelman also notes the 2006 study showed that class rank or grade 
point average (as an indicator of student effort) was as important as external senior tests (SAT or ACT) 
as measures of the academic resources students brought to bear on their postsecondary education. In its 
studies of the postsecondary trajectories of graduates of the Chicago Public Schools, the Consortium for 
Chicago School Research also found a high grade point average (GPA) mattered more than a high ACT 
score in determining access to any kind of college, to gaining admission to four-year versus two-year 
colleges and to more selective versus less selective colleges, and to succeeding in college. The study also 
found that nearly 60 percent of Chicago's graduating seniors had a C or D average, and that the GPAs 
of African-American and Latino students were significantly below those of white and Asian students.” 
A 2012 analysis of data from a statewide community college system found that high school GPA was 
strongly associated with both postsecondary GPA and credit accumulation.” 


Score well on the ACT or SAT tests: Both the College Board and ACT have conducted analyses of 
how well their tests predict future academic success, each setting benchmarks for college readiness. 
Based on these benchmarks, they each were able to determine how well different groups of students 
were prepared for the postsecondary challenges that awaited them. The College Board found fewer 
than half (43 percent) of 2011 college-bound seniors who had taken the SAT in 2011 met the Col- 
lege and Career Readiness Benchmarks that indicate a high likelihood of succeeding in college.” In 
addition, according to ACT: 


Of the 29 states where at least 40% of all 2011 high school graduates took the ACT, in only 1 state 
did more than half of the graduates meet at least three of the four College Readiness Benchmarks. 
In another 11 states, 40%-49% of graduates met three or four Benchmarks. In 12 of the 29 states, 
30%-39% of graduates met at least three of the four College Readiness Benchmarks in 2011, while 
less than 30% of graduates did so in 5 states. In no state did more than 55% of ACT-tested graduates 
meet three or four Benchmarks.”° 


Worse, disaggregating the data shows stark differences between different groups of students, as shown 
in figure 6. The results tell us that there is much work to be done. 


°6 Clifford Adelman, Moving Into Town—And Moving On: The Community College in the Lives of Traditional-age Stu- 
dents (Washington, DC: US. Department of Education, Office of Vocational and Adult Education, 2005), 119. 


7 Roderick, From High School to the Future, 2006, 43. 


*8 Belfield, Clive R. and Crosta, Peter M., Predicting Success in College: Predicting Success in College: The Importance of 
Placement Tests and High School Transcripts, Community College Research Center, CCRC Working Paper Number 42, 
February 2012, pp. 17-19. 


»” College Board. 43%, of 2011 College-Board Seniors, 2. 
39 ACT, Condition, slide 7. 
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Figure 6. Students meeting all four ACT College and Career Readiness Benchmarks 


Percent of students meeting all four 
College & Career Readiness Benchmarks 
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SOURCE: ACT: The Condition of College and Career Readiness, 2012, Based on Slide 5. 


What Questions Do We Need to Ask about College and Career Readiness? 


Given the findings from national studies of students’ postsecondary readiness, what questions should 
districts and individual schools be asking about how well they are preparing their students for postsec- 
ondary academic success? We offer the following questions as a place to start (see table 1). 


Table 1. Questions to ask about postsecondary readiness 


Readiness Leverage Points @]UT=S) (fe) ats) 


Taking the core curriculum classes | What classes are our students taking that will prepare them for 
that are generally recommended postsecondary success? 


as part of a pre-college curriculum. What proportion of our graduates has taken the core curriculum? 


Are there substantial differences in which students are taking the core 
curriculum? 


Maintaining a grade point average | What proportion of our students is earning a B or better grade point 
of B or better average? 


Are there substantial variations in the grades being earned by different 
groups of students? 


Taking additional higher- What advanced classes does our school offer? 
level classes, particularly in 


: - What advanced classes are our students taking? 
mathematics and science 


How well are students doing in these advanced classes? 
What proportion of our graduates has taken advanced level classes? 
Are there substantial differences in who enrolls in advanced classes? 


Are there substantial variations in how well different groups of students 
perform in advanced classes? 
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Readiness Leverage Points @]UT=S) (To) ats) 


Scoring well on the ACT or SAT How many of our students are taking college entry tests? Which ones are 
tests they taking (PSAT/SAT, PLAN/ACT)? 


How well are students doing on these tests? Are there substantial 
variations in which groups of students are taking college entry tests? 


Do the performance levels of our students indicate that they are meeting 
the benchmarks for college and career readiness? 


Are there substantial variations in the way different groups of students 
score on these tests? 


What do our students’ results on these tests tell us about where we need 
to improve? 


Postsecondary Access 
Six Leverage Points of Postsecondary Access 


Traditionally, guidance counselors helped students select and apply to postsecondary institutions in 
the junior and senior year of high school. More recently—especially in communities where students 
are likely to be the first in the family to get a postsecondary education—high schools are starting the 
process earlier. Through a series of studies on the postsecondary transitions of Chicago Public School 
(CPS) graduates, the Consortium for Chicago School Research has identified the following stages of 


gaining access to postsecondary education: 


e Aspire to postsecondary education or training (PSE/T). 
e Plan to apply to PSE/T. 

e Complete an application. 

e Apply to the right type of PSE/T. 

e Apply for financial aid (FAFSA). 

e Enroll in PSE/T. 


Aspire to postsecondary education or training (PSE/T): Many more students are increasingly aware they 
will need postsecondary education and training. According to ACT, 89 percent of tested students in 
2011 aspired to complete some level of postsecondary education.*! For more advantaged students 
with higher levels of parental education, postsecondary aspirations are shaped early, often well before 
they enter high school. However, first-generation students often do not know what postsecondary 
options exist and which steps they will need to take to get into a college or technical training program. 


Early exposure to postsecondary options has become more common in recent years. High school 
guidance curricula now usually include ninth grade sessions on planning for one’s educational future. 
These often include career interest inventories to help students understand the role their education 
will play in attaining their personal and occupational goals. Students’ early aspirations start the pro- 
cess of preparing for and gaining access to postsecondary options, but there are more steps to com- 
plete before a student enrolls in a college or a training program. 


Plan to apply to PSE/T: Planning to complete a postsecondary education involves multiple activities 
such as researching postsecondary options, visiting different schools or programs, signing up for a set 
of course work, and completing sample college application forms or writing sample college essays. 
With little knowledge of what it will take academically to qualify or how the application process 


31 ACT, Condition, slide 9. 
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works, students without strong support systems are less likely to plan for their future education and 
apply, secure financial aid, and enroll even when accepted. 


The gap between the proportion of students who aspire to a postsecondary degree or credential and 
those who attain one is substantial. Only 41 percent of Chicago seniors who stated that they aspired 
to complete a four-year degree actually applied to and enrolled in a four-year college.” 


Many postsecondary planning activities require the kind of adult support that is more often avail- 
able from parents in more affluent and education-savvy households than in economically challenged 
communities where few adults have completed a postsecondary education. To address this challenge, 
several states have launched postsecondary transition websites, such as What’s Next Illinois or RU- 
ReadyND in North Dakota, that provide students, parents, and educators a wide array of information 
and self-planning tools for career and college planning. These sites also collect data that allow districts 
and schools to monitor some aspects of the college preparation process, such as the completion of 
FAFSA forms. 


Complete an application to a postsecondary institution: Students in affluent families and communities 
often get help from parents and school or private counselors to help them through the complicated 
process of filling out college applications. For students without these supports, the process is intimi- 
dating. Some websites or software programs facilitate the process, but they are not always available to 
schools without cost. To increase the likelihood that more students will gain access to a postsecondary 
education, many schools try to integrate the process of completing applications into senior year Eng- 
lish or advisory classes and even include applying to college as a graduation requirement. 


Two aspects of the application process merit special attention: 1) matching students to institutions 
for which their academic credentials qualify them, and 2) filing financial aid forms early enough to 
secure scholarship assistance. 


Apply to an institution matched to one’s ability: Part of the choice of postsecondary institution involves 
matching students to the right kind for them. Longitudinal analyses of Chicago high school gradu- 
ates found that students are more likely to persist in postsecondary institutions when they enter more 
selective colleges with demanding academic expectations, but that large percentages of students en- 
rolled in colleges for which they were overqualified.*’ These findings were replicated by a study of five 
large school districts conducted by the Harvard Strategic Data Project.** A similar analysis in North 
Carolina found significant “undermatching.” It also found students from low-income or limited pa- 
rental education were more likely to apply to schools for which they were overqualified.** The Chi- 
cago researchers believe that, at least in some cases, many students who enrolled in two-year colleges 
did so less out of choice and more as a default solution in the absence of guidance and information.*° 


Apply for financial aid (FAFSA): Securing financial aid is a critical step for most students, but gaining 
access to these valuable resources involves lots of paperwork and deadlines. To qualify for aid at most 
institutions, students must file the Free Application for Federal Student Aid (FAFSA), which requires 
their parents to complete a complex form. In addition, since most postsecondary institutions use the 
FAFSA in their scholarship decisions and make those decisions in April, it is important that students 


w 
t 


? Roderick, From High School to the Future, 2008. 
Roderick, From High School to the Future, 2008. 

* Harvard Strategic Data Project, The College Match. 

*» Bowen, Chigos, and McPherson, Crossing the Finish Line. 
*° Roderick, From High School to the Future, 2008, 42. 


From Information to Action: A Guide to Using Postsecondary Data 18 


and their families complete the form by the end of March of senior year. A 2009 study found that 
students who received assistance completing the FAFSA and information about financial aid were 
40 percent more likely to apply for financial aid and 25 percent to 30 percent more likely to enroll in 
college.” 


Enroll in PSE/T: Even with considerable support along the pathway to postsecondary education, 
some students who have been accepted into college do not enroll in the year following graduation 
from high school. The Chicago Consortium studies of postsecondary transitions found that as many 
as 10 percent of accepted students did not enroll in college. In many cases, the reasons may be finan- 
cial. Students who did not file FAFSA applications may not have received financial aid or may have re- 
ceived insufficient assistance. Even with financial aid, the costs of attending college have risen steeply 
and, especially in an economically strapped family, the cost of foregoing income for a young person 
to pursue higher education can lead to family pressure (spoken and unspoken) for the young person 
to delay entry. A nationally representative survey of 1,600 dependent undergraduate college students 
and parents found that the average family spending on college declined in 2012 for the second year in 
a row, with more families reporting that they are making their college decisions based on the amount 
they can afford to pay.** 


Chicago now monitors each of these steps, as described in the profile below. 


Chicago Public Schools—Increasing Access to Postsecondary 
Education 


In 2003, in a concerted effort to increase access to postsecondary education, the Chi- 
cago Public Schools (CPS) created the Department of Postsecondary Education and 
Student Development. In doing so, it was the first large city school district to devote 
dedicated staff to helping students negotiate the transition from high school to col- 
lege. CPS was also was one of the first large districts to secure and use the National 
Student Clearinghouse data on postsecondary enrollment and persistence to guide 
its efforts. The Postsecondary Department also learned from the 2006 and 2008 re- 
ports analyzing the postsecondary trajectories of CPS graduates by the Consortium 
on Chicago School Research. Its mission was to design and implement an array of 
postsecondary, academic, financial, and social support programs and to build univer- 
sity, corporate, and civic partnerships designed to enhance college access. 


In 2009, CPS created the Office of College and Career Preparation (OCCP), which 
combined the Department of Postsecondary Education and Student Development 
with the Department of Education To Careers. OCCP joined a larger unit called 
the Office of Pathways to College and Career in 2011. OCCP staff includes approxi- 
mately 350 Academic Counselors and more than 40 College and Career Coaches 
who work at the school level, as well as a team of College and Career Specialists who 


focus on postsecondary preparation and work to support the school-based college 
and career coaches. OCCP also includes coaches for the Advancement Via Indi- 
vidual Determination (AVID) and Gaining Early Awareness and Readiness for Un- 
dergraduate Programs (GEAR UP) programs, and a team of Career and Technical 


* Eric P. Bettinger et al., The Role of Simplification and Information in College Decisions: Results From the H@R Block 
FAFSA Experiment (Cambridge, MA: National Bureau of Economic Research, 2009). 


8 Sallie Mae, How American Pays for College 2012: Sallie Mae’s National Study of College Students and Parents Conducted 
by Ipsos Public Affairs (Washington, DC: 2012). 
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Education (CTE) College and Career Academy Coordinators. All are working on 
helping CPS students chart pathways to postsecondary success. 


ChooseYourFuture.org 


OCCP operates a website called ChooseYourFuture.org, which gives students easy ac- 
cess to “the resources they need to plan a successful path to graduation and post-high 
school success.” 


‘The site includes a link to What’s Next Illinois (WNI) for postsecondary planning, a 
website operated by the Illinois Student Assistance Commission (ISAC), which has 
been working to increase access and affordability of postsecondary education for II- 
linois students. It offers students, parents, and educators a wide array of information 
and self-planning tools for career, high school, college, and financial aid planning. 
ISAC also allows districts and schools to monitor some aspects of the college prepa- 
ration process, such as completion of the FAFSA forms. 


ChooseYourFuture.org also provides a wide array of supports for staff to assist stu- 
dents and links to reports on CPS students’ FAFSA completion, college enrollment 
and retention, and employment. As these reports demonstrate, data play a key role in 
the work of OCCP, where the team monitors on-track performance attendance and 
behavior completion of CTE sequences test scores college applications percentage of 
applications where students’ qualifications are appropriately matched to the selective- 
ness of the institution FAFSA completion rates college acceptances scholarship dol- 
lars received enrollment and persistence in postsecondary education 


The CPS FAFSA initiative 


The Consortium on Chicago School Research’s 2008 report found that, among CPS 
students who had been accepted to a four-year college, those who completed the 
FAFSA were 50 percent more likely to enroll than those who had not. More than 
one-third of the students who were accepted to a four-year college, but did not com- 
plete FAFSA, did not enroll at any postsecondary institution. Low-income students 
who do enroll in postsecondary education but do not complete the FAFSA pay con- 
siderably more for their education than those who file FAFSA forms, which compro- 
mises their ability to complete their education. 


Submitting the FAFSA early in the calendar year is also critical to maximizing the amount 
of financial aid available to students from low-income families as most states and postsec- 
ondary institutions require aid applications to be submitted on or before April 1. Begin- 
ning in 2007, the CPS Department of Postsecondary Education and Student Develop- 
ment used data provided by the Illinois Student Assistance Commission to track FAFSA 


completion among its seniors, disseminating the data regularly to schools, and providing 
resources to support schools in helping students complete the FAFSA. Between 2007 and 
2010, the CPS program increased the share of students submitting FAFSA forms from 
64.5 to 86.4 percent. In 2010, by completing FAFSA forms, CPS students qualified for 


more than $100 million in federal and state financial assistance.*” 


» Chicago Public Schools, “CPS students post record FAFSA completion rate,” press release, February 28, 2011. 
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Monitoring the match 


Through OCCP, CPS now monitors the extent to which students’ postsecondary 
applications are matched to their academic qualifications. In addition, in partnership 
with the nonprofit organization MDRC, CPS is piloting a College Match interven- 
tion in three CPS high schools. This program is based on the National College Advis- 
ing Corps, based at the University of North Carolina. 


“Beginning in the spring semester of a student’s junior year and extending through 
high school graduation, College Match delivers services through a combination 


of workshops and counseling by trained advisors who are recent college graduates 
themselves. These advisors replicate what is available to help students in more affluent 
schools and homes make informed decisions about college enrollment.” 


Early results found that students served by the College Match program chose to at- 
tend more selective colleges and universities than students in the comparison group.” 


Chicago's efforts to increase postsecondary enrollment have paid off. As shown in the 
table below, there has been more than a 12% gain and 28% increase in postsecondary 
enrollment for CPS students. Moreover, the increases in postsecondary enrollment 
for African-American and Latino students have been even greater. 


Table 3: Percent of CPS graduates enrolled in postsecondary education 


2004 yA0) OF) 2006 2007 2008 yA0} 0) 2010 (ore Tia) increase 


Overall 43.5% | 46.0% | 47.9% | 50.0% | 52.5% | 54.4% | 55.7% | 12.2% 
al el lien held 
6. 7. : 5 


7. 6 64. 7. 70. 13. 
African 42.8% | 46.7% | 47.5% | 50.2% | 53.7% | 54.9% | 56.7% | 13.9% 
American 

4. 39 3. 6. 8. 3. 


SOURCE: Data from Annual reports of Chicago Public School Graduates Enrolled in Postsecond- 
ary Education based on National Student Clearinghouse data, including cohorts from 2004 to 2010. 
Reports downloaded from ChooseYourFuture.org. 


What Questions Do We Need to Ask about Postsecondary Access? 


As the gap remains wide between students who aspire to a postsecondary education and those who 
actually complete a degree or certificate, what questions should districts and individual schools be 
asking about how well they are helping their students gain access to postsecondary options? Table 2 
offers the following questions as a place to begin. 


“0 Jay Sherwin, “Make Me a Match: Helping Low-Income and First-Generation Students Make Good College Choices” 
(MDRC Policy Brief, April 2012). 
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Table 2. Questions to ask about postsecondary access 


Access Leverage Points 


Aspire to postsecondary education 
or training (PSE/T). 


Plan to apply to PSE/T. 


Complete an application to PSE/T. 


Apply to PSE/T well-matched to 


their academic qualifications. 


Apply for financial aid (FAFSA). 


Enroll in PSE/T. 


Questions 


How many of our students aspire to go on to postsecondary education 
and training? 


Are there substantial differences between students who do and do not 
aspire to postsecondary education? 


Are there substantial variations between different groups of students 
in the kinds of postsecondary institutions and programs they aspire to 
attend? 


What activities and services does our school provide to help students 
plan for their postsecondary future? 


What steps do our students take to learn about their postsecondary 
options? 

How many students have completed career interest inventories to help 
them plan their educational futures? 

How many students have visited postsecondary institutions? 


How many students have taken courses or attended programs on a 
college campus? 


Are there substantial variations between different groups of students 
in the number and kind of steps they take to plan their postsecondary 
education? 


How many students who aspire to postsecondary education apply to one 
or more postsecondary choices? 


Are there substantial variations in the number of applications submitted 
by different groups of students? 


What proportion of our students applies to institutions for which they are 
qualified? 

Are there substantial differences in which groups are more likely to apply 
to postsecondary institutions that match their academic qualifications? 
How many students applied for financial assistance? 

How many students filed FAFSA forms before 3/31? 


Are there substantial differences in which groups of students are more or 
less likely to apply for financial aid? 


What impact does filing for financial aid have on applying to 
postsecondary institutions? 


What impact do filing FAFSA forms have on our students enrolling, if 
accepted? 


How many students were accepted to postsecondary institutions or 
programs? 


What proportion of these students enrolled in postsecondary institutions 
or programs in the year following graduation? 


Are there substantial differences in which groups of students are more 
or less likely to enroll in postsecondary institutions? What factors are 
associated with students not enrolling in the postsecondary institutions to 
which they have been accepted? 
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Postsecondary Success 
Four Indicators of Postsecondary Success 


Using Adelman’s research as well as other studies of postsecondary outcomes, we can identify critical 
indicators of postsecondary success that reflect, at least in part,"' the effectiveness of efforts to prepare 
students to succeed at the postsecondary level. These indicators vary depending on whether students 
attend two-year or four-year colleges. In addition, there is more complete information about post- 
secondary trajectories of students attending four-year institutions than for those attending two-year 
institutions and programs. This reflects the wider range of students who attend two-year institutions 
and the broader array of programs, degrees, and certificates at these institutions. Still, there are four 
important markers along the way to success: 


Take credit-bearing courses in the first year of postsecondary studies. 


e Accumulate 20 credits. 
e Complete a second year of studies. 
e Complete a degree or certificate. 


Take credit-bearing courses in the first year of postsecondary studies: At both the two-year and four-year 
level, entering directly into credit-bearing course work without needing to complete remedial or de- 
velopmental work is an important predictor of eventual success. Research has consistently shown that 
students who need to take significant numbers of noncredit courses are less likely to complete their 
postsecondary studies.** Noncredit course work still costs students money, using up often-limited 
resources for postsecondary education. In addition, students can become discouraged as they take 
general courses without taking beginning classes in their chosen field of study. This indicator also 
reflects most directly the quality of preparation. 


Accumulate 20 credits: The threshold of 20 postsecondary credits appears to be an important predictor 
of completion for both two-year and four-year students. Research more clearly defines the time frame 
within which these credits need to be earned at the four-year level, where reaching this benchmark in 
the first calendar year of study (including summer school, if needed) is associated with higher levels of 
degree completion.* Because of the greater variation in student ages and attendance patterns at two-year 
colleges, the time frame is variable, but the 20-credit benchmark still predicts success, particularly for 
younger community-college students.“ Passing credit-bearing mathematics course work appears to have 
particular importance at the two-year college level, where passing the second credit-bearing math course 


“| The Consortium for Chicago School Research found significant variations in the graduation rates of similarly selective 
four-year colleges. See Melissa Roderick et al., From High School to the Future: A First Look At Chicago Public School 
Graduates’ College Enrollment, College Preparation, and Graduation from Four-Year Colleges (Chicago: Consortium on 
Chicago School Research, 2006), 72-77. 

© For four-year colleges, see Clifford Adelman, Answers in the Toolbox, Academic Intensity, Attendance Patterns, and 

Bachelor’s Degree Attainment (Washington, DC: US. Department of Education, 1999). For two-year colleges, see Paul 

A. Attewell et al., “New evidence on college remediation, Journal of Higher Education 77 (2006), cited in Thomas 

Bailey and Sung-Woo Cho, “Developmental Education in Community Colleges,’ (Columbia University Teachers Col- 

lege: Community College Research Center, 2012). 


® Alexander C. McCormick, Credit Production and Progress Toward the Bachelor’s Degree: An Analysis of Postsecondary 
Transcripts for Beginning Students at 4-Year Institutions (Washington, DC: US. Department of Education, National 
Center for Education Statistics, 1999). 


“ Clifford Adelman, Moving Into Town—and Moving On: The Community College in the Lives of Traditional-age Stu- 
dents (Washington, DC: US. Department of Education, 2005). See also Juan Carlos Calcagno et al., “Stepping Stones 
to a Degree: The Impact of Enrollment Pathways and Milestones on Community College Student Outcomes,’ Com- 
munity College Research Center Working Paper 4 (October 2006): 21. 
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is a predictor both of earning an associate's degree and transferring to a four-year institution. 


Complete a second year of studies: Researchers now talk about the importance of credit momentum 
toward earning a degree at both the two-year and four-year college level. Many four-year college stu- 
dents leave school during their first year of postsecondary studies, but a substantial additional group 
that return for a second year never gain enough academic traction——either in terms of credit accumu- 
lation or GPA—to complete a degree. At the four-year college level, finishing a second year of studies 
is a better predictor of degree completion than simply returning for a second year. At the two-year 
college level, researchers examined the relationship between specific milestones, such as completing 
25 percent, 50 percent, and 75 percent of the program, and finishing a degree or certificate. As one 
study found, “reaching milestones associated with credits earned or completing a portion of a pro- 
gram increased the probability of graduation for all students, but doing so had a greater impact on 
younger students than on older ones.’” 


Complete a degree or certificate: The final indicator of success in postsecondary education is comple- 
tion of a diploma or certificate. As shown in figure 7, completion has significant economic conse- 
quences for students. 


Figure 7. Lifetime earnings by educational attainment level 
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SOURCE: Anthony P. Carnevale, Nicole Smith, and Jeff Strohl, Help Wanted: Projections of Jobs and Education Require- 
ments Through 2018 (Washington, DC: Georgetown Center on Education and the Workforce, 2010). 


What Questions Do We Need to Ask about Postsecondary Success? 


Table 3 suggests questions about postsecondary success that can be used to reflect on how well high 
schools have prepared their students. It is important to recall that differences in how well different 
postsecondary institutions retain students with similar academic characteristics make it hard to at- 
tribute postsecondary outcomes beyond the first year to high school preparation. 


Adelman, Ibid. See also Colleen Moor and Nancy Shulock, Student Progress Toward Degree Completion: Lessons From 
the Research Literature (Sacramento, CA: Institute for Higher Education Leadership and Policy, 2009), 4. 


46 Adelman, The Toolbox Revisited, 2006, 55-57. 


“” Calcagno, Stepping Stones, 5. 
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Table 3. Questions to ask about postsecondary success 


Success Leverage Points 


Take credit-bearing courses. 


Accumulate 20 credits. 


Complete a second year at 
four-year colleges or 50% of 
their program requirement at 
two-year institutions. 


Complete a diploma or 
certificate. 


Questions 


How many of our students enter directly into credit-bearing classes at the 
postsecondary level? 


What proportion of our students needs to take remedial classes in 
mathematics? 


What proportion of our students needs to take remedial classes in English? 


Are there substantial variations between the levels of remedial work needed 
by different groups of students? 


What proportion of our graduates who enter postsecondary education 
accumulates 20 credits by the end of their first year of postsecondary study? 


Are there substantial differences between groups of students in whether or 
not they accumulate 20 credits by the end of their first year of postsecondary 
study? 


What high school factors are associated with students not accumulating 20 
credits by the end of their first year of postsecondary study? 


What proportion of our graduates who go on to postsecondary education 
completes a second full year at the four year colleges or 50% of their 
program requirement at two-year institutions? 


Are there substantial variations between different groups of students in 
whether or not they complete a full second year of studies or 50% of their 
program? 


What high school factors are associated with students not completing a 
second full year of course work or 50% of their program requirement? 


What proportion of our graduates earns a postsecondary degree or 
certificate? 


Are there substantial variations between groups of students in who did or did 
not earn a degree or certificate? 


What proportion of our graduates earns four-year college degrees? 
What proportion of our graduates earns two-year degrees or certificates? 


Are there substantial differences in which students are earning two- and 
four-year degrees or certificates? 


The following chapter discusses the public sources of data for answering questions about postsecond- 
ary readiness, access, and success. 
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CHAPTER THREE 


Sources of Data for Answering 
Questions about Postsecondary 
Readiness, Access, and Success 


This chapter describes the three major public sources of data for answering questions about postsec- 
ondary readiness, access, and success: state longitudinal data systems (SLDS), the National Student 
Clearinghouse, and district-level data systems. Each offers a different slice of the data collected on 
student performance that may be used to better understand how students perform after they graduate 
from high school. Data from these systems may be used in different combinations depending on the 
attributes and limitations of the systems in each state. 


State Longitudinal Data Systems 


In 2002, Congress passed the Education Sciences Reform Act and the Educational Technical Assis- 
tance Act. This legislation initiated an ambitious effort to establish comprehensive state longitudinal 
data systems (SLDS) in the states. These systems combine information from multiple sources for each 
individual student, allowing their performance to be studied across time, schools, and institutional 
levels. 


“A few pioneering states—including Florida and Texas—had begun similar efforts spanning all educa- 
tion sectors from kindergarten through postsecondary education prior to 2002. However, the new 
laws had the objective of providing resources and guidance for building such systems to all the states 
and territories. Beginning in 2005, the U.S. Department of Education administered five separate 
rounds of grants intended to build and improve SLDS. So far, grants have been awarded to 47 states 
and the District of Columbia.” 


Also, it is important to note that every state has committed to the objectives associated with SLDS 
development as a condition of receiving State Fiscal Stabilization Funding under the American Re- 
covery and Reinvestment Act. Thus, the effort is nationwide. 


The national SLDS program has five primary goals——to promote the use of education information 
in states to: 
e improve instruction 


e determine whether graduates have the knowledge and skills to succeed in postsecondary edu- 
cation and the workforce 


e facilitate the transparency of education information for stakeholders through public reporting 
e ensure the use of data to inform decision-making at all education levels 


e generate and facilitate the use of accurate and timely data 


The intent was to realize these goals in two ways. The first involved connecting detailed student-level 
data from preschool, K12 schools, and districts with postsecondary and workforce data. The second 
anticipated providing regular releases of information through reports, research findings, evaluation 
study results, and other mechanisms. 
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Initially, much of the focus was on building comprehensive, longitudinal K12 data systems at the state 
level. Developing K12 longitudinal data and reporting capabilities has remained a core objective in 
each round of grants, but later rounds also placed a priority on linking K12 data to postsecondary 
education and workforce data. 


Developing the SLDS is a daunting undertaking, but most states have made substantial progress in 
building the kinds of systems anticipated by the legislation. Most states have SLDS systems that are 
operating while continuing to add new data sources and build capacity to provide information and 
insights to education policy makers and practitioners. The specific content and capabilities of the sys- 
tems differ from state to state. Because the development processes are relatively recent and rapid, it is 
important to note SLDS content and capabilities are still changing and emerging. 


As a condition of receiving flexibility around specific requirements of the Elementary and Secondary 
Education Act (ESEA), states must annually report college enrollment and credit accumulation rates by 
high school. Eleven states (CO, FL, GA, IN, KY, MA, MN, NJ, NM, OK, and TN) made this commit- 
ment when they submitted requests through the first round of ESEA waivers in November 2011. 


Kentucky’s State Reports 


Kentucky is one state that provides feedback reports on some aspects of postsecondary readiness. 
Each high school now receives a four-page report for each graduating cohort with the following in- 
formation: 


Performance of graduates in comparison with other high schools in the district and state 


e Number of graduates 
e Average GPA 
e Percent Free and Reduced-Price Lunch 


e Average eleventh grade ACT scores of these students in English, Reading, Mathematics, and 
Science, and the composite across tests 


e Dollars awarded to graduates in Kentucky Education Excellence Scholarships 


e Percent of graduates attending postsecondary institutions (from Kentucky postsecondary in- 
stitutions and National Student Clearinghouse data) 


e Percent of graduates attending Kentucky institutions 
e Percent of students attending out-of-state institutions 


Profile of students attending postsecondary institutions 


e Number of students attending in-state or independent colleges 

e Percent attending different types of institutions: four-year public, two-year public, independent 
e Percent of students enrolled full-time 

e Percent seeking different types of degrees (bachelor’s, associate’s, certificate, undeclared) 


e Demographics of students enrolled in postsecondary education by race/ethnicity, gender, 
Special Education, free and reduced-price lunch 


Types of institutions attended by graduates 


e Percent attending 


= — In-state four-year and two-year public, and proprietary 
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* Independent institutions (both in and out of state) 
" Out-of-state proprietary 
= Not attending postsecondary 
e Average composite eleventh grade ACT score by type of postsecondary education 


In addition, the report provides a list of the institutions attended by graduates. 


Indiana’ State Reports 


Indiana also provides each high school with a three-page College Readiness Report that compares its perfor- 
mance for each graduating cohort to other high schools in the state and includes the following information: 


e Number of graduates in the cohort attending Indiana Public Colleges 
e Percent of graduates needing remediation at the postsecondary level 
e Percent of graduates needing remediation by subject in Math, English Language Arts, or both 


e Number and percent needing remediation by different types of diplomas: honors, Core 40 
(Indiana's college preparatory curriculum), general or unknown 


e Postsecondary enrollment by the type of degree sought: bachelor’s, associate’s, certificate (un- 
der one-year programs versus at least one year but less than two-year programs) 


e Percent enrolled as full-time and part-time students 


e Number and percent enrolled by institution types: two-year Indiana public and four-year 
Indiana public institutions 


e Number and percent enrollment by program types: education; arts and humanities; social 
and behavioral sciences and human services; science, technology, engineering, and math; 
business and communication; health; trades, and undecided 


e Average freshman year cumulative GPA by diploma type: honors, Core 40, general and other 
or unknown 


e Average credit hours earned by institution type and full- or part-time status 


Both states’ reports give individual schools valuable insights about their students’ postsecondary 
progress. Because the Indiana high school feedback reports include information on enrollment in 
remediation classes, high schools in this state can see how well their graduates are performing at the 
postsecondary level.** Samples of both feedback reports are included in Appendix A. 


Current SLDS Content and Capacity 


The SLDS are a valuable resource for high schools as they try to chart a pathway to postsecondary suc- 
cess for their students. However, because the SLDS are still developing, printed materials about what 
each collects and makes available may soon be outdated. 


Given the rapid development of the SLDS, no single resource provides information about the re- 
al-time status of each of the states’ systems. The Education Department’s web page for the SLDS 
state grants provides state-by-state information and the names of the key state contacts. (http://nces. 
ed.gov/programs/slds/grant_information.asp) 


‘8 Because ACT has studied the relationship between scores on the ACT exams and placement in remedial classes at the 
postsecondary level, Kentucky high schools also can get a rough estimate of how many and what proportion of their 
students are likely to need remediation by looking at their ACT scores. 


From Information to Action: A Guide to Using Postsecondary Data 28 


Those SLDS that are mature are most likely able to provide nearly all of the longitudinal data (span- 
ning 2002-2012, for example) required for analytic work. In some states, the data will be housed in 
a single data repository such as a data warehouse. In other states where data collected by the SLDS 
are less extensive, data may be available through interagency data-sharing. Importantly, the SLDS will 
have in place all of the matching processes and algorithms necessary to combine district-level data 
with other data resources, such as those from postsecondary education. In addition, SLDS managers 
will have templates for the necessary agreements or contracts required to access the data at a single 
place and through a single process. In general, these kinds of agreements should be forged between 
the district and the SLDS, rather than by individual schools. 


In the best of all worlds, the SLDS will provide access to most, if not all, of the requested data. How- 
ever, given their varying stages of development, not all SLDS will be able to meet all school/district 
requirements. To enable schools to work with SLDS data, district data personnel should establish an 
early relationship with the managing entity for the SLDS in their state. Usually, this is a state orga- 
nization such as a state education agency, but sometimes is a university, an education data research 
center, or a “K16” organization with SLDS responsibilities that spans secondary and postsecondary 
data sources. Whether or not the SLDS can meet all of the site’s data requirements, data experts at the 
state education agency may be able to provide valuable services, guidance, and facilitation with regard 
to accessing data resources. 


Moreover, because the SLDS are still in development, the expressed data needs of districts and schools 
provide valuable information for SLDS managers. Most try to anticipate the kinds of analyses that 
will be most useful, but as they contemplate future design work and content considerations for the 
SLDS, the kinds of data requests they receive will inform their thinking. 


Working with the SLDS 


Because the SLDS are continuing to mature, several issues should be kept in mind in obtaining and 
using data: 


1. Access. Most states have established processes that provide an avenue for external organiza- 
tions to request and gain access to student-level data from the SLDS. This can be expected to 
increase as SLDS access processes will be accelerated by work currently being done by the In- 
stitute for Education Sciences (IES), the research arm of the U.S. Department of Education.” 


District data personnel should determine early on what the access processes are, the time 
frames involved, information needed in requests and agreements, and state expectations re- 
garding managing and securing data resources. In addition, as in all cases involving access to 
student-level data, requirements and obligations regarding the handling of the data will be 
outlined in some form of a legal agreement or memorandum of understanding (see the discus- 
sion of FERPA and framing data requests in Chapter Six). 


2. Longitudinal Content. To examine the trajectory of students beyond graduation, high schools 
need several years of data——at the very least enough to look back at the last two years of high 
school and forward through the first two years of postsecondary education. To meet this 
requirement, an SLDS would need to contain longitudinal data spanning at least those four 
years. While much of the SLDS developments in states began with federal grants in 2005, not 


# National Forum on Education Statistics, Forum Guide to Supporting Data Access for Researchers: A State Education 
Agency Perspective (NFES 2012-809) (Washington, DC: National Center for Education Statistics, US. Department 
of Education, 2012). IES worked with state education agencies through the NCES Forum during 2011 to develop the 
guide, which outlines considerations, processes, and templates for states to use in establishing consistent access policies 
and processes. Further, the Regional Education Laboratories have been reconstituted by IES, and much of their work 
will focus on using state longitudinal data systems to fulfill their research agenda. 
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all SLDS will have data covering the full period. However, all SLDS are designed with the ex- 
pectation of continuous data collection going forward, and many can already provide useful 
longitudinal perspectives on the relationship between high school preparation and postsec- 
ondary performance and persistence. 


3. High School Data Content. In the best-case scenario, the SLDS will have the same high school 
data in similar detail to what is available locally. These data will already be matched over time, 
in some cases with postsecondary data. Many SLDS already have the capacity to provide these 
data, but other states are still building their capacity to capture vital elements for examining 
postsecondary readiness and success. 


This does not mean that these elements are not available elsewhere. For example, schools op- 
erating in states that do not collect student-level course completion or transcript data may be 
able to find those data elsewhere. They should check with their state education agency (SEA) 
to determine if these data are collected in some form at the state level but are not yet incor- 


porated into the SLDS. 


In some cases, the state system may participate in a transcript exchange service that collects 
and transfers this type of information from high schools to selected postsecondary institu- 
tions. The repository that facilitates this exchange may provide access to transcripts for local 
research purposes. 


If this information is not available at the SEA, local school district data managers usually col- 
lect course-level data asa part of their district’s management information and student support 
functions. Another alternative, although more labor-intensive and complicated, is to link or 
match data obtained from the local district containing course completion data to SLDS data 
provided to the school. 


4, Postsecondary Data Content. Most states now “have the ability” to link K12 data to postsec- 
ondary data. This is good news from the perspective of using SLDS resources to answer ques- 
tions about postsecondary success. Keep in mind, however, that this postsecondary content 
may not include all postsecondary institutions nor all desired data elements, such as: 


Postsecondary Coverage. The term “postsecondary” covers a broad range of options. The op- 
tions include private-for-profit, private-nonprofit (often referred to as “independent”), and 
public institutions. They include opportunities inside a state’s boundaries as well as outside 
them. Within these categories, many states have an array of postsecondary technical institu- 
tions with programs of varying length, two-year colleges, four-year colleges and universities, 
and postgraduate institutions. 


When states indicate that their SLDS has the “ability to link to postsecondary,’ they are likely 
referring to most but not necessarily all of the postsecondary options. In most cases, the SLDS 
includes data from in-state, public, two- and four-year institutions. In some cases, they also 
may include data from independent and proprietary institutions. For many schools, coverage 
that includes public two-year and four-year institutions within their state will be sufficient 
because it will cover the postsecondary experiences of most of their students. 


Postsecondary Data Elements. When a state indicates that its SLDS links K12 and postsec- 
ondary data, this may include a range of data elements. Discussions with state SLDS con- 
tact persons will be needed to determine if the types of data elements needed by a school or 
district, such as postsecondary course completion, remediation, and overall completion, are 
captured or can be captured through the SLDS. In addition, as the SLDS are developing, it 
is important to determine whether the SLDS contains the needed data elements over a suf- 
ficiently long period of time to track graduates forward into postsecondary and through to 
completion. 
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It is likely that all of the desired postsecondary data elements may not be available in the 
SLDS. However, state-level postsecondary governance organizations” have detailed elements 
that can be provided either through an arrangement with the SLDS or through a separate ar- 
rangement.”! 


National Student Clearinghouse Data 


The capacity of most SLDS to provide information about postsecondary outcomes is limited to the 
institutions within each state. In addition, the SLDS sometimes do not include students who attend 
proprietary institutions. Another major resource for obtaining matched student-level high school 
data to postsecondary data is the National Student Clearinghouse (NSC). The NSC collects stu- 
dent-level postsecondary enrollment and related data from postsecondary institutions throughout 
the United States. It was originally established in 1993 as an enrollment verification mechanism for 
lenders providing loans to students enrolled in postsecondary education as authorized under Title IV 
of the Higher Education Opportunities Act. Since its inception, the NSC has expanded its services to 
include additional reports and analyses of use to high schools and postsecondary institutions. 


Currently, over 3,000 postsecondary institutions throughout the country participate voluntarily in 
the NSC. The NSC’s website™ indicates that participating institutions account for 96 percent of the 
postsecondary enrollments in the United States. Participating institutions report data into the NSC 
on an average of every 45 days. 


The primary service of interest to high schools is called StudentTracker, which includes data on the 
postsecondary enrollment, persistence, and completion of the school’s graduates. Another service, 
DegreeVerify, enables high schools to verify degree and certificate completion. Statistical reports pro- 
vided by NSC include a variety of high school feedback-types of analysis describing many aspects of 
student transition from high school to postsecondary education.* The detailed student-level matched 
datasets” include the following information: 


e Identifiers—First Name, Middle Initial, Last Name (obtained from the high school) 


e Student Characteristics—Birth Date (from the high school) 


e — Institution— College Code/Branch (OPE, FICE), College Name, State, two-year/four-year/ 
less than two-year public/private (from StudentTIracker) 


e Enrollment—Begin/End date, status as full-time, half-time, less than half-time, leave-of-ab- 
sence, withdrawn, deceased (from Student Tracker) 


e Persistence—A derived measure based on matches across several defined enrollment periods 


(from StudentIracker) 


°° Examples of postsecondary governance organizations include higher education coordinating boards, higher education 
commissions, boards of governors or regents, university of community college departments, and others. In some cases, 
coordinating boards and commissions collect most of the state’s postsecondary data; in other cases, the data may be 
collected by several organizations. 


>! A good summary resource for state systems is Strong Foundations: The State of State Postsecondary Data Systems: 2012 
Update on Data Sharing with K-12 and Labor, by the State Higher Education Executive Officers organization. The 
report, based on survey results, is available at http://www.sheeo.org/sspds/default.htm. 


~ hetp://www.studentclearinghouse.org/about/who_we_work_with.php 


* Sample high school reports are available at http://www.studentclearinghouse.org/high_schools/files/STHS_Sam- 
pleReport.pdf 


** See format at http://www.studentclearinghouse.org/high_schools/files/STHS_DetailReportGuide.pdf. 
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e Graduation—Anticipated graduation date from StudentTracker, reported graduation from 
DegreeVerify 


High schools and postsecondary institutions can avail themselves of NSC services by entering into 
an agreement designed to comply with FERPA requirements and by paying associated fees.” With 
agreements in place and fees paid, high schools participate by providing secure files containing indi- 
vidual student data elements, such as cohorts of graduates, to the NSC. The NSC then matches these 
students to the postsecondary data they have collected. Based on matched data, the NSC provides a 
variety of statistical reports as well as individual-level records. 


The matching processes used by the NSC rely on matches between first and last names, middle ini- 
tials, and birth dates. Their processes include a physical review of questionable matches such as those 
with partial information. As NSC works with institutions to collect other data elements, such as gen- 
der, additional processes will be explored to improve match accuracy. The NSC continues to collect 
more information as well as offer new or expanded services. 


District-Level Data Sources 


Almost all school districts now have some kind of electronic data system. They typically include sepa- 
rate systems that have developed over time to collect and report data, often with the capacity to inte- 
grate and share data files across the original systems. Figure 8 shows the most frequently used systems. 


Figure 8. Types of electronic student data systems 


Student information systems provide real-time access to student data such as attendance, demographics, 
test scores, grades, and schedules. 


Data warehouses are electronic data collection and storage systems that provide access to current and 
historical data on students, personnel, finance, and other factors. 


Instructional or curriculum management systems provide a unifying framework to support access to 
curriculum and instructional resources such as planning tools, model lesson plans, creation of benchmark 
assessments, linkage to state content or performance standards, and communication and collaboration tools 
(e.g., threaded discussion forums). 


Assessment systems support rapid organization and analysis of benchmark assessment data. 


SOURCE: Jeffrey C. Wayman, “Involving teachers in data-driven decision making: Using Computer data systems to sup- 
port teacher inquiry and reflection,” Journal of Education for Students Place At Risk 10, (2005): 295-308, cited in Barbara 
Means, Christine Padilla, and Larry Gallagher, Use of Education Data at the Local Level: From Accountability to Instruc- 
tional Improvement, prepared for the U.S. Department of Education, 2010. 


However, the results of a 2008 survey of more than 500 districts indicate some challenges remain 
to using district-level data. Despite multiple systems, most districts still face difficulty getting their 
systems to communicate with one another. More importantly for the focus of this guide, according 
to one study, “there is currently a misalignment between the types of data that districts feel are key 
for improving student achievement and the types of data that are being requested by the state for ac- 
countability purposes. Districts need student-level information to inform instruction and what is sent 
to the state is aggregated student achievement data, attendance, and student counts.” The extent to 
which this limitation will pose problems for efforts to understand and improve postsecondary readi- 
ness and success will depend on the kind of individual student data the district maintains and on the 


» hetp://www.studentclearinghouse.org/colleges/fees.php 


6 Means, Padilla, and Gallagher, Use of Education Data. 
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capacity of the district’s data office to share the data in meaningful and teacher-friendly ways. 


Matching Datasets 


The great advantage of longitudinal data systems is that they make it possible to follow individual stu- 
dents over time and across institutions. When longitudinal data are not available, individual student 
records can be matched across different data sets. Although this is sometimes described as a relatively 
simple process that uses a common identification number (where it exists), accurate matches require 
more precision. 


As an example of the complexity of linking and matching, the Florida Department of Education man- 
ages the nation’s longest-lived longitudinal data repository that joins secondary and postsecondary edu- 
cation data sources to other data. This repository contains data linking individual student records from 
K12, adult education, career and technical education, and associate’s and bachelor’s degree programs 
from 1996 forward. To ensure accurate matches across sectors and over time, the system uses various 
combinations of name, birth date, gender, school-assigned ID, and social security number with business 
rules designed to ensure reliably accurate matches.” The processes include physical reviews used to re- 
solve questionable matches and to improve the matching processes defined as business rules. 


Underlying business rules establish processes that deal with reporting anomalies that occur between 
systems and over time, including issues such as inconsistently reported data elements, changed ele- 
ments, and elements that may no longer be reported. 


We would discourage the development and use of matching and linking methodologies by schools or 
districts unless they are essential to the analyses being conducted and there are no other reasonable 
alternatives. Instead, we suggest using different sources of data to get a larger picture of student post- 
secondary success that drill down to potential ways to improve results. 


The next chapter will discuss working with local postsecondary institutions to gather more fine- 
grained data on graduates. 


* The business rules are defined in a Florida K-20 Education Data Warehouse Meta data application at http://edwapp. 
doe.state.fl.us/home.aspx. 
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CHAPTER FOUR 


Collecting Data from Postsecondary 
Partners 


Researchers have learned that enrollment in remedial course work is an indicator of weak postsecond- 
ary readiness and, conversely, a negative predictor of future postsecondary success. But individual 
student data about this phenomenon have been difficult for high schools to acquire. Enrollment in 
remedial classes is not included in NSC data, and is only recently becoming available in some states’ 
postsecondary databases. Yet this element, more than most others, is the one that most directly re- 
flects back to high schools how well they have prepared their graduates for the rigors of postsecondary 
preparation. 


High schools now can learn from SLDS or NSC data where their graduates have enrolled in postsec- 
ondary programs. Because many students attend institutions close to home, high schools and school 
districts can develop strategic partnerships with the top receivers of their graduates to learn more 
about the quality of their preparation. They also can build programmatic bridges to address weak- 
nesses that compromise students’ futures. 


This chapter discusses the potential of working with the postsecondary institutions that receive large 
shares of a high school’s graduates. It includes profiles of two different approaches to working with 
postsecondary data. 


Tapping into Resources within Reach 


In many communities, students choose to attend a local community college, university or techni- 
cal training institute. This decision is often reinforced by proximity to home, familiarity, assumed 
cost savings, local traditions, and sometimes a lack of proactive planning to go elsewhere. Once high 
schools identify which postsecondary institutions have enrolled large shares of their graduates, they 
can reach across the K12/higher education divide to create partnerships. These cross-sector relation- 
ships may help both levels discuss and align academic expectations and create feedback loops that, in 
turn, lead to improvement in students’ preparation for success at the next level. 


High schools can acquire some data from local institutions by getting reports from their SLDS or 
from the NSC. These sources will let them know which institutions students attend, whether they 
attend part-time or full-time, whether they continue attending in successive years, and whether they 
have graduated. However, these data only give a partial picture of students’ postsecondary status. The 
data points do not provide adequate information about gaps in students’ readiness and thus are not 
ones that high school teachers can easily see as within their reach to affect 


Through a local partnership, both high schools and postsecondary institutions can get more granu- 
lar, personalized, and nuanced data that, in turn, can help them better understand their students’ 
academic strengths and challenges. These data can be most helpful in shaping new thinking and new 
practice. Here are a few examples of potential data points that postsecondary institutions can share 


with their high school feeders: 


e Enrollment in postsecondary remedial or developmental course work 
e Postsecondary GPA 


e Postsecondary credit accumulation 
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e Postsecondary persistence 
e Financial aid 
e High School GPA and transcript information 


Using these data, the partners can learn: 


e Howhigh school GPA correlates with enrollment in postsecondary remedial classes 
e Howhigh school GPA correlates with postsecondary credit accumulation 


e Whether specific courses taken in high school (e.g., advanced mathematics, Advanced Place- 
ment) are associated with higher levels of postsecondary success 


e Whether there are varying patterns of success among different racial, ethnic, economic, or 
language groups 

e Whether completing FAFSA forms and receiving student postsecondary financial aid are as- 
sociated with enrollment and persistence 


New York City Partnership 


The partnership between the New York City Department of Education (NYCDOE) and the City 
University of New York (CUNY) is one example of a local arrangment. Through the negotiation of 
cross-institutional data agreements, longitudinal data records for individual students are now being 
used to understand and improve postsecondary success. 


Longitudinal Data-Sharing across Institutional Boundaries: 


New York City Public Schools and City University of New York 


The City University serves 262,000 academic credit students and 269,808 adult, 
continuing, and professional education students. Almost 70 percent of all current 
CUNY students are graduates of New York City high schools. The CUNY system 
encompasses 23 separate institutions including six two-year community colleges and 
11 four-year colleges, as well as a graduate school. The CUNY student population is 
very broad and includes many low-income and first-generation college students. 


In 2008, the New York City DOE and CUNY forged a partnership that focuses on 
improving the postsecondary outcomes for all students in the city. At the core of the 
partnership is a unique data-sharing agreement that allows the two institutions to 
match students and track them over time by exchanging student-record-level data. 


The joint data agreement enables NYCDOE to go beyond the data available for students 
enrolled at CUNY through the National Student Clearinghouse, allowing the district to 
collect and report on finer grain information such as which courses students have taken in 
college and how well they performed. Currently, the district is able to collect the following 
information on their students who have enrolled in the CUNY system: 


Admissions Data 


e Choice of college (when applying students rank CUNY colleges in order of 
preference) 


Demographics, including gender, date of birth, ethnicity, place of birth and na- 
tive language 
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e SAT scores 

e High school credits by subject 

e Scores by subject on the New York State Regents examinations 

Initial Student Proficiency 

e Whether a student required remediation in reading, writing, or math when 
entering CUNY 


Scores on CUNY’ basic skills exams in reading, writing, and math for those 
students who are not already exempt from remediation because of their Regents 
or SAT scores 


Enrollment History 


e Dates of semesters enrolled 

e Credits attempted and earned 
e Majors 

Course History 


Course subject and title 
Credits attempted and earned 
Remedial courses attempted and passed 
Grade per course 
Degrees 
Degree level 
Cumulative GPA 
Graduation date 


Major upon graduation 


So far, two major outcomes have resulted from the data-sharing work. 


“Where Are They Now” Reports. Using the CUNY data, NYCDOE has created user- 
friendly reports entitled Where Are They Now? The reports, given to each individual 
high school principal, show the outcomes of different cohorts of their students in the 
CUNY system. Following the graduates for two years to report on their progress, and 
comparing them to overall NYCDOE results, each report includes: 


e Number and proportion of a school’s graduates who enrolled at CUNY in the 
fall semester following graduation and the type of program (AA/BA) in which 
they are enrolled 


Percentage of graduates attending CUNY who required remediation in reading, 
writing, or math 


Persistence of graduates enrolled at CUNY over the first four semesters, sepa- 
rated by whether or not they needed remediation; 


GPA and credits attempted, and credits accumulated, of graduates, separated by 
whether or not they needed remediation; 
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Persistence of graduates enrolled at CUNY over the first four semesters, sepa- 
rated by whether they were enrolled in a BA or AA program; 


GPA and credits attempted, and credits accumulated, by graduates, separated by 
whether they were enrolled in a BA or AA program. 


After the initial distribution of the reports, NYCDOE solicited input from the 
schools on how to make the reports more user-friendly and then made major en- 
hancements. Now, rather than giving high schools static PDF documents with their 
data, NYCDOE produces dynamic reports that can be manipulated and disaggre- 
gated by school staff. The reports have dashboard pages and tabs that allow schools 
to toggle between different cohorts and subgroups, and they also allow schools to see 
trends over time. 


New School Report Cards. Each school in New York City receives an annual report 
card that reports on its school climate, student performance, and student progress. 
To increase the focus on postsecondary outcomes, NYCDOE has placed three new 
postsecondary data points into the high school report cards: 


e The college preparatory course index, which measures the percentage of stu- 
dents in the cohort that entered high school four year earlier who have met at 
least one benchmark for college readiness;** 


The college readiness index, which measures the percentage of students who 
entered high school four years earlier who graduated and met the requirements 
for exempting from remedial coursework at CUNY;” 


The college enrollment rate, which measures the percentage of students who 
entered high school five years earlier who graduated and enrolled in a degree 
program at a two-year or four-year college by the end of the fall semester follow- 
ing their graduation. 


Starting next year, these data points will be figured into the school report card grades 
and fully integrated into the accountability system in the district. 


A School-level Approach When SLDS Data are Not Available 


The example below describes a school-level approach to working with postsecondary data when the 
desired longitudinal data are not available. At the core, however, the goal is the same: to improve the 
performance of the school’s graduates in postsecondary education. It is important to note that this 
latter approach usually requires building a working relationship with an individual or group of indi- 
viduals at the postsecondary level who can be helpful in shepherding the work of getting the remedial 
course data. If the postsecondary institutions have an office that addresses the problems of students 
with weak academic skills, this may be the best place to start. 


*8 These include scoring 65+ on the Algebra II, Math B, Chemistry or Physics Regents exam, scoring 3+ on any Advanced 
Placement Exam, scoring 4+ on any International Baccalaureate exam, or earning a C or higher in a course for college 
credit. 


» Students can exempt from remedial coursework by reaching score thresholds on the NY State Regents examinations, 
the SAT, and/or course standards accepted by CUNY. 
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Using Data to Foster Curiosity and Engagement in Improving 
Postsecondary Readiness: Casa Grande High School and 
Sonoma State College 


In Petaluma, California, Casa Grande High School’s leaders have a pivotal school goal: 
to increase the number of college-ready students who graduate able to meet the Califor- 
nia university system’s A-G entry requirements“ and able to succeed in college without 
remediation. The central role played by postsecondary data is reflected in the notes from 
collaborative teacher planning meetings in the school, which has been a Smaller Learn- 
ing Communities Program (SLCP) grantee twice. Teachers ask one another, What can 
postsecondary data sources tell us about the effectiveness of our teaching? Which data 
sources should we use? How can we best help our seniors move from “conditional” to 
“college-ready” in math? To what extent do our students’ results on final course exams 
correlate with state tests? How well do our students do at college? 


School leaders at Casa Grande want the term “data-driven decision making” to be 
what they do, not just what they say they want to do. They have used their profession- 
al learning communities or PLCs (initially developed through their SLCP grants) as 
the vehicle to drive this way of integrating data into their everyday work. Teachers are 
organized into departmental PLCs as well as ninth- and tenth-grade Houses and four 
eleventh- and twelfth-grade clusters: Green Careers Pathway; Health Careers Path- 
way; Liberal Studies; and Marketing, Media and Management. PLC conversations 
occur during weekly meetings lasting one to three hours that sometimes voluntarily 
spill over to after school and weekend hours. 


Through the PLC conversations, sobering schoolwide data can be cut reduced to a 
manageable size. Finding out that students earned 150 F’s in core academic classes 
can be daunting. But realizing that those grades were received by 85 individual stu- 
dents and only 25 to 30 of those students are in a specific SLC can make the problem 
manageable. When names and faces replace the numbers, teachers can resist a sense 
of futility and be moved to action. 


Casa Grande leaders know that data are only discrete bits of information, but they 
can pack a punch depending on the way they are presented and what they mean to 
individual teachers. Through trial and error, school administrators and teachers have 
learned to thoughtfully organize any data to be discussed, to know the purpose of the 
data discussion at hand, and to consider the impact from multiple perspectives. 


A school data report card serves as the foundation for analysis at Casa Grande. Over 
time, school leaders have learned that less is more: they have selected a few high lever- 
age data points from ninth through twelfth grade and from postsecondary sources. 
They have learned what data generate automatic resistance from staff, as well as what 
points encourage staff engagement and a desire to respond proactively. 


What data cause negative reactions? Examples include state achievement tests requir- 
ing substantial class time for preparation and administration that are measured in the 
spring and reported in the fall when teachers no longer have contact with the stu- 
dents. Data summaries “done to” teachers, presented urgently under the assumption 


® The A-G requirements include the sequence of high school courses needed to satisfy minimum eligibility requirements 
for the University of California and California State University systems 
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that teachers need to be prodded to improve, are not helpful to them. 


What data prompt curiosity and a willingness to make adjustments? Data from the 
test administered under the California Early Assessment Program, measuring readi- 
ness for college reading and math, as described below: 


The Early Assessment Program (EAP) is a collaborative effort among the 
State Board of Education (SBE), the California Department of Education 
(CDE) and the California State University (CSU). The program was estab- 
lished to provide opportunities for students to measure their readiness for 
college-level English and mathematics in their junior year of high school, and 
to facilitate opportunities for them to improve their skills during their senior 
year.” 


The EAP tests are augmented California Standards Tests (CSTs) in 11th- 
grade English and mathematics; these tests are part of California’s public 
school testing and accountability system and are required of all students. The 
augmented tests were developed by CSU and K-12 faculty, who made sure 
that both the California high school standards as well as the CSU placement 
standards were covered. The faculty added a writing sample to the English 
CST, as well as a few more test items, but they kept the time needed for test- 
ing to a minimum.” 


Case Grande staff obtain the data early enough to absorb their meaning, take steps 
to help students meet postsecondary requirements, and discuss ways to improve what 
and how they are teaching to get better results. They also can download the data files 
in order to slice and dice the data by department and small learning community. 


At Casa Grande High School, the Mathematics Department Chair, who also sits on 
the SLC Leadership Team, spearheads the school’s EAP initiative. He coordinates 
with a data person at the Sonoma County Office of Education who is able to query 
the data; he can then compare those results with school math test performance and 
course-taking patterns. 


The established staff credibility earned by the department chair, combined with his 
willingness to publicly share his professional concerns about his students’ lack of col- 
lege readiness, offers a powerful example of a teacher using test data to adapt and 
differentiate his lessons and course designs. His leadership and humble inquiry are 
paving the way for the English department to do likewise. 


Working with the EAP data has led to partnerships with Sonoma State College and 
Santa Rosa Junior College, the two institutions that receive the majority of Casa 


Grande’s graduates. A faculty member from Sonoma State also has come to speak 
to Casa Grande’s juniors. She encourages them to take the EAP, which is voluntary, 
explaining how much money they will save if they enter directly into credit-bearing 
courses instead of remedial classes. Over the past three years, EAP participation has 
held steady at 99 percent for English, and participation in Mathematics has jumped 


*! Definition downloaded from http://www.calstate.edu/eap/. This website includes public access to school and district 
test results, private student access to their test results and an analysis of their areas of weakness, as well as support for 
both students and teachers. 


© Description downloaded from http://www.calstate.edu/eap/documents/eap_program_description.pdf. 
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from 59 percent to 100 percent. 


Not only do the EAP data let students know their college preparedness, but also they 
challenge faculty to reassess the rigor of their courses. Casa Grande cannot link the 
students’ records across the secondary and postsecondary divide, but they do their 
best to look at multiple sources of data over the trajectory of students’ academic ca- 
reers. Working with their postsecondary partners, faculty and administrators exam- 
ine data from the remedial classes students are taking at Sonoma State College. They 
also look at other California state test data from the California Standards Test (CST) 
and the High School Exit Examination (CAHSEE), disaggregating the data for all 
the school’s subgroups. 


When the high school and its local postsecondary partners meet, the presentation of data 
is formal, but not uniform. Staff and leadership alike are trying to identify what data sets 
are the most informative and likely to impact practice, and they continue to study at what 
point in the school calendar the release of data is most likely to drive action. 


These efforts are slowly bearing fruit. Although on the rise, the EAP data indicating col- 
lege readiness are still alarmingly low, hovering around 31 percent in both English and 
Mathematics. The postsecondary data have enabled teachers to become deeply engaged 
in efforts to improve their students’ readiness. As the Casa Grande assistant principal 
explained, “For teachers to change, they have to see a tangible connection between their 
high school work and what kids do in college. They can feel their impact in seeing stu- 
dents able to enter directly into credit-bearing classes and save money.” 


The next chapter will discuss some additional ways to collect school-level data that can be useful in 
understanding a range of factors involved in students’ preparation for postsecondary success. 
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CHAPTER FIVE 


Using Surveys and Focus Groups 


Administrative data from state longitudinal data systems, the National Student Clearinghouse, and 
district databases can provide valuable guidance about the patterns of achievement and difficulty 
among different groups of students. Practitioners can form hypotheses about the factors that produce 
the patterns they see in the data and then gather school-based information to test these hypotheses 
and provide additional insights that can help guide strategies for improvement. Surveys and focus 
groups can ask questions that help identify and illuminate underlying issues that are associated with 
these patterns. 


Using Surveys to Gain More Insight about Patterns in the Data 


As with any form of inquiry, it is critical to start by focusing on the questions that most need to be 
answered. What problems or dilemmas exist in your understanding of how well students—both as a 
whole and within particular groups—are doing in achieving their postsecondary aspirations? Then, 
who should be surveyed to find answers to these questions? 


Most high schools administer some form of pre-graduation survey to students to find out their future 
plans. Some conduct follow-up phone or mail-in surveys of graduates, asking where they went after 
high school. According to one high school administrator, this has become easier in the age of the cell 
phone, as students often retain their cell phone numbers even after leaving home. 


Research in recent years has increased understanding of the kinds of academic knowledge and skills 
needed to succeed at the postsecondary level. It also has confirmed the importance of a college-going 
culture and helped define the critical supports needed to increase both aspirations and access to the 
most appropriate postsecondary options. However, the typical pre-graduation surveys miss a valuable 
opportunity to ask some carefully chosen questions about how well students are prepared for post- 
secondary success. 


A survey administered to seniors can ask about the extent to which they have been challenged to do 
the kind of academic work that helps them prepare for success at the next level and about the supports 
they have received when they encountered academic difficulty. Survey questions also can solicit infor- 
mation about the extent to which students have experienced the elements of a college-going culture 
or received critical guidance to help them select the right postsecondary options and course work or 
to help them apply for financial assistance. 


In designing the survey instrument, it is important to be strategic in limiting the number of ques- 
tions asked and to resist the temptation to seek too much information. School staff should begin by 
identifying what they believe is most important to learn first and from whom. With this in mind, 
they can develop questions or seek expert assistance to design the questions for any survey to ensure 
ease of data gathering and analysis.® Sites can then use free or low-cost tools, such as Zoomerang or 
SurveyMonkey, to create online, readily accessible instruments for students to fill out. For the most 
honest responses, students should be assured that their answers will be confidential, and the amount 
of identifying information students are asked to provide should be kept to a minimum. 


° The Consortium for Chicago School Research provides examples of past student surveys on its website (http://ccsr. 
uchicago.edu/content/index.php); these include several surveys with questions for high school seniors. In some cases, 
CCSR notes that permission to use these questions must be requested. 
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What Kinds of Survey Questions Are Helpful in Understanding Postsecondary Preparation? 


Student surveys provide a good opportunity for asking students how different actors in the school 
(teachers, counselors, administrators, and peers) shaped their thinking about what to do after gradu- 
ation. For example, survey questions can ask about the following. 
e ‘The messages student hear in school about the importance of postsecondary education: 

= from teachers in classes 

= from counselors 

* in informal interactions with school personnel 

= from peers 


e The extent to which students perceive that teachers expect all students to pursue postsecond- 
ary education options 


e The extent to which teachers and counselors at the school have helped students plan for their 
future in such areas as: 


"career inventories and educational requirements of different career options 
* advice about the range of available career and college options 

7 guidance about choosing appropriate postsecondary options 

* — college-exploration activities (materials, websites, speakers, visits) 


" concrete assistance in negotiating the application process and filling out financial aid 
forms 


Survey questions also can ask about the kind of academic preparation students have experienced. 
e ‘The extent to which students have had the kinds of challenging academic experiences that 
prepare them for postsecondary academic work, such as: 
* taking a college preparatory course load 
=" writing research Papers using external sources of information 


" taking advanced classes (honors, Advanced Placement, International Baccalaureate) or 
college courses 


= attending college programs in the summer 


e The extent to which students have been able to get support from teachers that helped them 
succeed and take on more difficult course work 


e The extent to which students have developed critical learning behaviors and study skills 


The analysis of the survey data need not be complicated. Little statistical sophistication is needed to 
get a picture of how widespread these experiences are in the senior class. For a somewhat more fine- 
grained analysis that can identify differences in the extent to which groups of students have the same 
experiences and perceptions, the survey should include some basic descriptive questions about the 
student (gender, race/ethnicity, general level of academic performance™). 


“ Te is difficult to include socioeconomic status in surveys—usually measured by asking about eligibility for free and 


reduced-price lunch—~as adolescents often are hesitant to reveal this information and sometimes do not know their 
free-lunch eligibility or economic status. 
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Conducting Focus Groups to Tap Student Perceptions 


Students often have valuable insights about which aspects of high school preparation truly helped or 
hindered them. These insights may not be evident in their responses to survey questions and may need 
to be collected through other means. Focus groups often are used to help illuminate the patterns that 
surface from analyses of quantitative data. Data collected through focus groups also can surface new 
questions that can be tested through survey and other quantitative data. 


Designing and Conducting Effective Focus Groups 


Purpose: Focus groups are interactive, unpredictable, engaging, and variable. A focus group can gen- 
erate interesting information that is unlikely to surface through other means. It can offer insights into 
ideas that can be followed up in more detailed interviews or document reviews. As such, focus group 
data are most valid when triangulated with data collected from other sources. The primary uses for 
focus groups as a data source include: problem identification, planning, implementation, and assess- 
ment.® 


Table 4. Characteristics of a focus group 


What A Focus Group Is... What a Focus Group Is Not... 

A qualitative research method best used in conjunction A source of information that can be generalized as 
with other data gathering techniques also true for others 

A guided group discussion eliciting participants’ A go-around where each participant must answer 
experiences and beliefs each question 


Focused conversation with a series of planned questions | A free-for-all conversation about a topic 


Dependent on participants’ willingness to trust in the Required Q and Awhere participants are 
process and share relevant information compelled to respond 

An opportunity for the facilitator to listen carefully and Achance for the facilitator to share personal 
clarify opinions or beliefs 


Structure and Process: In planning a focus group, it is important to clarify the goal and purpose of the 
planned focus groups, which will in turn help determine: 

e ‘The degree of structure in the conversation 

e Howto recruit participants for the focus group and who should be represented 

e How many and which participants should be in each group and how long it should last 

e ‘The number of groups to be conducted 

e Logistics (room set-up, food, etc.) 

e Confidentiality procedures, and 

e ‘The contents of the opening statement 
Importance of the Opening Statement: The opening statement of a focus group is critical in setting the 
stage for the conversation and building an environment of openness and trust so that participants feel 
comfortable sharing their perspectives. Standard components of the opening statement include: 

e Welcome and appreciation for participation 

e Purpose of the focus group discussion 


e What will be done with the records of the conversation (notes and tape) 


® David L. Morgan and Richard A. Krueger, The Focus Group Kit (Thousand Oaks, CA: Sage Publications, 1998). 
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e Level of confidentiality guaranteed to participants 

e Role of the facilitator 

e What to expect as the conversation proceeds (time and process) 

e Ground rules 

e Request for participants’ afirmation to abide by ground rules and proceed. 


‘The cycle of questions: Focus groups follow a cycle that begins by engaging the participants and then 
move through an introduction into the core areas of inquiry. The table below shows the cycle of ques- 
tions. (A script for the opening statement and suggestions for an initial activity to encourage engage- 
ment is included at the end of this chapter.) 


Table 5. The cycle of focus group questions 


Question Type Purpose 

Opening Participants get acquainted and feel connected 

Introductory Begins discussion of topic 

Transition Moves smoothly and seamlessly into key questions 

Key Obtains insight on areas of central concern in the inquiry 

Ending Helps researchers determine where to place emphasis and brings closure to the discussion 


It is important to ask questions that are concrete and not abstract. Participants should be queried for 
their personal experiences rather than general opinions. Questions should be open-ended and invite 
elaboration, without asking for explanations. When giving examples, it is useful to ask for positive and 
negative examples separately, beginning with positive ones. 


Examples of some classic focus group questions: 
e What do you expect to do after graduating from high school? Who in this school talks with 
you about your future plans? 
e Hasanyone talked to you about what it takes to get to college? What is involved in getting into college? 


e Do you know anyone who goes to college? What does it take to get into college? What does 
it take to succeed in college? 


e Howdo you decide whether you are “college material”? What are some ways that you can tell 
that you are capable of going to college? 


e Have you thought about paying for college? 


e How have you made your decisions about where you want to study after high school? 


Adaptation of Focus Group Protocols for Less Formal Group Conversations 


In some cases, less formal group conversations may be useful in gathering individual stories from per- 
sonal experiences and thoughts of students, teachers, and guidance staff. These conversations can be 
a productive way of collecting evidence of strategies that support student achievement and insights/ 
opinions about weak postsecondary preparation. 


Recommendations for adapting typical focus group protocols to structured small group conversa- 
tions include: 


e Size: Six participants maximum per group is ideal, but if a larger group appears they would 
not be turned away, and facilitators can adapt to a group interview format. 
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e Time: Short enough to fit within a class period to minimize school-day disruptions. 
e Introduction: Clarification of purpose, process, confidentiality, and use of results. 
e Number of questions: five to six maximum, due to shortened length of each focus group. 


e Content: Relevant, concrete, open-ended questions that ask the perspective of each partici- 
pant group. Questions should encourage the telling of stories and real experiences. Ask “what” 
and probe for clarification and detail; they should resist asking “why.” 


e Process: A short, factual go-round at the start lets each participant speak; a brief paper and 
pencil strategy deepens engagement, especially for students. The goal is to seek facts, exam- 
ples, and stories rather than engage in opinion-driven discussions or arguments. 


e Flexibility: The facilitator can either run all communication through him/herself (like the 
hub of a wheel) or encourage intra-group conversation where participants react directly to 
one another’s statements. 


There are different approaches to analyzing focus group data. The first involves reading carefully 
through the notes to pull out the recurring themes and examples of them in the conversations. A sec- 
ond approach after having conducted multiple focus groups on the same topic is more quantitative. 
Here, after pulling out the main themes, the occurrences of similar statements can be counted to get 
a sense of the frequency of a topic—e.g., teachers’ comments about continuing education after high 
school—that has taken place during the conversations. 


Sample Opening Statement for Focus Groups 


Welcome and thanks for your willingness to help us learn more about what your high 
school is doing or can do in the future to improve students’ success in college. This in- 
formation will help us give feedback to the leadership of this school and other schools 


like it. 


I will take notes so that we can accurately remember your comments when we write 
our summaries. I will keep the notes and will not share them with others. I will never 
use your name or connect you with your comments; I will guarantee that this conver- 
sation is confidential. This conversation will not be anonymous since there are oth- 
ers who are aware that you are participating. Again, the information from this focus 
group will not be shared with anyone else in your school. 


I will ask a series of questions. Not everyone will answer each question. Sometimes I 
may directly invite you to respond. You may always pass. I will try to make sure every- 
one can join in. If you've already made some comments, I may ask you to pass and let 
others participate. 


Our focus group will last about XX minutes. 


Ground Rules: Please let one person speak at a time. Speak from your own personal 
experience. You are not representing others or a group. Let’s show respect for differ- 
ent views. It is OK to differ. And, please refrain from using any names when telling a 
story — say “a teacher” or “a friend,” or something like that. 


Can everyone agree to these guidelines? 


Is everyone still willing to participate? 
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OK, let’s begin. 
Opening Strategies to Engage Participants 


One way to get participants engaged is to begin by asking them to do a brief writing 
exercise and then share their results. 


Ask participants to jot down three adjectives about X (something related to the 
topic). 

Mark an X on a continuum and explain why location of item is chosen. 
Complete a sentence about X. 


Make an analogy about X. 


Make notes about a positive or negative experience that exemplifies X. 


The next chapter discusses how to begin to answer these questions. It will focus on framing the ques- 
tions, identifying what kind of data are needed, and requesting the data from the common sources of 
longitudinal student data. 
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CHAPTER SIX 


Framing Questions and Collecting Data 


This chapter begins by discussing the process of developing questions for inquiry about postsecondary 
readiness, access, and success. It will then provide practical advice on the process of collecting data, 
including how to determine what data are needed and the procedures needed to receive the data. 


Step One: Frame the Questions and Identify the Data Needed to Answer Them 


The first step in planning a quest for data from any source is to identify what data elements are need- 
ed—a process that begins by framing the essential questions that are to be answered. Using data to 
inform the learning process is part of a cycle of inquiry, as shown in figure 9. 


Figure 9. The cycle of collaborative inquiry 


Framing the 
Question 


Monitoring Collecting 
Results Data 


Drawing 


Conclusions Analyzing 
Taking data 


Action 


Organizing 
Data Driven 


Dialogue 


NOTE: Based on Nancy Love, Using Data/Getting Results: A Practical Guide for School Improvement in Mathematics and 
Science (Norwood, MA: Christopher-Gordon Publishers, Inc., 2002), p. 32. 


It is best to start with a simple, clearly focused question that helps narrow down the data you will 
need to consult. The research described in Chapter Two provides guidance about the elements that 
may be most useful as the focus of inquiry. Here are a few examples of questions that emerge from the 
research on postsecondary readiness, access, and success. 


Postsecondary readiness: Based on research that has shown higher levels of postsecondary success for 
students who have passed higher levels of secondary mathematics, the following questions could 


be asked: 
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e Which advanced mathematics courses have our students taken? 
e Which students are taking advanced mathematics and science classes? 


e Whatare the differences between students who have taken advanced mathematics classes and 
those who have not? 


Postsecondary access: Based on research showing that attending postsecondary institutions for which 
a student is overqualified is linked to lower levels of postsecondary success, the following questions 


could be asked: 
e Where do our students enroll in college? 
e Do our students attend educational institutions that are matched to their qualifications? 


e What are the differences between students whose choices match their qualifications and 
those whose choices do not? 


Postsecondary success: Given the research showing that the ability to enroll directly in credit-bearing 
course work is a major factor associated with postsecondary persistence and success, the following 
questions might be posed: 


e What percentage of our graduates needs to take remedial or developmental classes at the 
postsecondary level? 

e What percentage of our graduates earns 20 credits in their first year of postsecondary study? 

e What are the differences between students who earn 20 credits and those who do not? 


e Once the questions are determined, you will compile a list of the kinds of data elements need- 
ed from secondary and postsecondary sources to answer them. 


Table 6 provides examples of data elements needed to answer these questions for a specific cohort of 
students (e.g., students who entered ninth grade in 2009). 
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Table 6. Examples of data elements 


Data Element 


Individual Identifiers 


Student Characteristics 


School 


Test Data 


Course Data 


Acceleration Course Data 


“Placement” Test Scores 


Synopsis 

State Assigned ID 
Name 

Birth Date 

Social Security Number 


Home Address 

Gender 

Race/Ethnicity 
Free/Reduced Lunch Status 
Limited English Speaking 
Disability Status 


High School (s) 

Middle School (s) 

NCLB summative tests 

Course name & taxonomic reference 
Course Grade 


Dual or concurrent enrollment 


Advanced Placement 


International Baccalaureate 
ACT 
SAT 


Common Placement tests 


Student Graduation Data Diploma Type 
Award Date 


Postsecondary Enrollment 


Postsecondary 
Remediation 


Postsecondary Credit 
Accumulation 


Postsecondary Graduation 


Postsecondary Institution Type 
Postsecondary Institution Name 


Term date first entered 


Term designation 

Remedial English/Math/Both 
Postsecondary Persistence | Persistence second term 
Postsecondary Courses Course name and taxonomic reference 


Degree Type (associate’s, bachelor’s, certificate, 
employer certification) 


Degree Major 
Award Date 
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Likely Source of Data 
SLDS 
District Data System 


SLDS 
District Data System 


SLDS 

District Data System 
SLDS 

District Data System 
SLDS 

District Data System 
SLDS 

District Data 


SLDS 
District Data 


SLDS 
District Data 
SLDS 


National Student 
Clearinghouse (NSC) 


SLDS 

Local Postsecondary 
SLDS 

Local Postsecondary 
SLDS 

NSC 

SLDS 

Local Postsecondary 
SLDS 

NSC 
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Step Two: Define the Cohort for Which You Will Request Data 


After defining which data elements are needed to answer your questions, the next step is to define a 
cohort of students whose data will be collected. This requires thinking about the breadth of informa- 
tion needed going “backwards and forwards.” You will need to have data that goes back far enough to 
capture the data points in high school that your questions require and forward enough for the post- 
secondary data system to have data on the outcomes you are interested in studying. 


For example, if your questions involve how high school performance in advanced mathematics of 
recent graduates is related to postsecondary performance, you will need to follow a set of previous 
graduates backward at least two years to capture data from the years during which they would have 
taken advanced mathematics classes, and forward at least two years into their postsecondary educa- 
tion to see whether they needed to take remedial classes and how many credits they accumulated in 
the first year of postsecondary study. 


On the other hand, if your inquiry concerns how well all students from your high school fared at the 
postsecondary level, you will need data that goes back to the beginning of their high school career to cap- 
ture the full cohort. In addition, you will need data going forward enough for their graduation outcomes 
to have entered the postsecondary data systems—as much as four years after high school graduation. 


Simply stated, the choice of a cohort beginning with 2012 high school graduates would provide little 
insight into postsecondary success in 2012. An optimal choice would be those students who gradu- 
ated from high school in 2006 (those who were in the ninth grade in 2002) and who are likely to 
graduate from college by 2012. The K12 data system would need to be able to provide four years of 
course-taking data “backwards” and the postsecondary resource would need to provide six years of 
data going “forward.” In the best case, the objective will be to build a complete high school course- 
taking and related performance record for each student during his or her high school career. 


An additional step may be required that involves translating the names of the data elements you need 
into a form that is recognizable as those from whom you are requesting data. In most cases, the “trans- 
lation” will depend upon the data source. Although there will be some similarities, each one is likely 
to have its own conventions and definitions, 


Most sources will have a “data dictionary” or a “meta data application” that provides data element 
definitions, naming conventions, reporting protocols and values, and a data element number. In most 
cases, the relationship between the types of data elements listed in table 10 and the elements identified 
in the data dictionaries will be straightforward. 


To ensure that the desired elements are clearly defined, it is recommended that you take the additional 
step of translating the data elements you need into the form recommended in the Common Educa- 
tion Data Standards. 


Step Three: Request the Data 


The next step in the process is requesting the data from the appropriate source, such as the NSC, 


6° The Common Education Data Standards (CEDS) process (see http://ceds.ed.gov/elements.aspx) has defined hun- 
dreds of data elements typical of secondary and postsecondary data systems. They were developed in collaboration with 
school districts, state education agencies, postsecondary institutions, state postsecondary systems, and national groups 
such as the Chief State School Officers, the American Association of Community Colleges, and the State Higher Edu- 
cation Executive Officers organization. While national, state, and local systems do not necessarily conform to the tax- 
onomies contained in CEDS, CEDS is broadly recognized and provides a common, agreed-upon language for the type 
of work discussed here. The latest version of the standards has included more depth about adult education and CTE in 
addition to K12 and postsecondary, as well as elements related to employment and earnings. 
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state data organizations, school districts, and postsecondary institutions. The process for making re- 
quests varies from state to state and district to district, so it is important to check to see what each 
recommends as the process for requesting and receiving data. In addition, the request will need to be 
consistent with the protections required by the Family Educational Rights and Privacy Act (FERPA). 


The Family Educational Rights and Privacy Act 


The Family Educational Rights and Privacy Act, known best as FERPA (20 USC 1232g), is a federal 
law that protects the privacy interests of parents and students in a student's “education records.” FER- 
PA affords parents the right to have access to their children’s education records, the right to seek to 
have the records amended, and the right to consent to the disclosure of personally identifiable infor- 
mation (PII) from education records, except as provided by law. The law applies to educational agen- 
cies and institutions that receive funds under any program administered by the Secretary of Educa- 
tion. This generally includes schools, school districts, colleges, and universities where students attend. 


Under FERPA, a parent or eligible student (i.e., a student who turns 18 years old or enters a postsec- 
ondary institution at any age) must provide a signed and dated written consent before the educational 
agency or institution discloses education records or PII from education records. FERPA includes ex- 
ceptions to the general consent requirement which are set forth in § 99.31 of the FERPA regulations. 
The exception that is particularly relevant to this guide pertains to the disclosure of information from 
education records to “organizations conducting studies for, or on behalf of, educational agencies or 
institutions for the purpose of developing, validating, or administering predictive tests, administering 
student aid programs, and improving instruction.” “ This type of disclosure is permissible without 
requiring prior written consent if the study is conducted in accordance with the requirements in § 
99.31(a)(6) of the FERPA regulations, including meeting all the requirements for written agreement 
under the studies exception. (Links to additional resources for addressing FERPA are included at the 


end of this chapter.) 


It is important to recognize that every organization that collects and manages education data derived 
from education records will need to implement data-sharing procedures that address the confidenti- 
ality and security provisions of FERPA. These procedures may vary according to the way local edu- 
cational agency officials choose to implement the requirements under FERPA and its regulations, as 
well as the policies and laws governing their own particular agencies. 


The protective procedures need to be evident in data request applications and data exchange agree- 
ments, contracts, or memoranda of understanding. The common thread of the content in the agree- 
ments will be that they are intended to ensure compliance with the FERPA requirements in § 99.31 (a) 
(6)(iii)(C). The common elements that need to be addressed in a written agreement or contract are 
as follows: 


1) The agreement must specify the purpose of the study, describe the scope and duration of the 
study, and the information to be disclosed. 


2) The agreement must specify that the PII from education records must only be used for the 
study identified in the agreement. 


3) There must be a requirement that the organization conduct the study in a manner that limits 
access to PII from education records only to representatives of the organization with legiti- 
mate interests. 


4) There must be provisions for destroying all personally identifiable information when the in- 


* SLDS Technical Brief, Guidance for Statewide Longitudinal Data Systems (SLDS) (NCES 2011-601, November 2010), 
4, 
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formation is no longer needed for the purposes for which the study was conducted and the 
time period specified in which the information must be destroyed. 


Ensuring Data Security 


As a potential recipient of confidential student data, the local school and district teams will need to 
establish security procedures designed to protect the data from inadvertent release.® The team should 
expect that it will have to describe its security processes as part of working with data providers on 
data-sharing agreements, covering three kinds of procedures: 


1) Technical Procedures: These include establishing firewalls, network and physical security, 
data storage protocols, password processes, and secure data destruction processes. 


2) Nontechnical Procedures: These define who can access and view what data, where the data 
can be accessed, and under what conditions. 


3) Publication Procedures: These define the processes for reviewing data products to ensure in- 
dividual identities are not inadvertently released because of small cell sizes or separate data 
displays that can be combined to reveal identities. 


In some cases, the team may choose to establish and manage its own data security procedures; in oth- 
ers, they may work with an existing facility that has well-established FERPA-compliant procedures, 
such as their school district or a postsecondary institution data center. 


Finally, it is important to allocate sufficient time and resources to develop the data request and negoti- 
ate the procedures needed to receive it. Once established, future data requests usually require less time 
and effort. 


The next chapter discusses an approach to working with the data that can help practitioners better 
understand their situation and weigh options for action. 


ADDITIONAL RESOURCES FOR ADDRESSING FERPA 


Recent revisions to FERPA Regulations in December 2011 (34 CFR Part 99) provide new and clarifying 
requirements related to SLDS systems and sharing data between secondary and postsecondary data 
resources. The December 2, 2011, regulations implementing FERPA are available at http://www2.ed.gov/ 
policy/gen/reg/ferpa/index.html. Other information and resources are available on the Family Policy and 
Compliance Office website at http://www2.ed.gov/policy/gen/guid/fpco/index.html). 


The U.S. Department of Education has established a Privacy Technical Assistance Center (PTAC) to act 

as the “one stop” resource for education stakeholders to learn about data privacy, confidentiality, and 
security practices related to student-level longitudinal data systems. PTAC provides timely information and 
[0] ofef=1¢=roe Ul (el-lalex=Melam o)ah\e-(eVAmexe)ani(ol=\alit=liiNam=laremcx=Le101 a1 Wal 0) ¢=\e1t(ex=\-Mm tg] colU(e] alr=(\c-1a(=10/ae) Mm c=\<(olU|cex-tseml are [6le||ale| 
training materials and opportunities to receive direct assistance with privacy, security, and confidentiality of 
longitudinal data systems. Information about the resources and tools are available from http://ptac.ed.gov/. 


68 A synopsis of procedures in available from the Privacy Technical Assistance Center at http://www2.ed.gov/ 
policy/gen/guid/ptac/technical-briefs.html. 
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CHAPTER SEVEN 


Working with the Data 


“Despite the increased amounts of data available, many educators still feel ill prepared to analyze and use 
their school data effectively. They are data rich but information poor.” 


The current focus on the power of data sometimes appears to assume that data constitute a self- ac- 
tualizing tool. Datasets themselves do nothing. People improve schools; datasets inform the people, 
helping them make smarter—and at times difficult—decisions. The educators deciphering the data 
and receiving the messages within them must subsequently take action. A clear understanding of the 
meaning of a data set (or combination of them) can help practitioners understand their challenges 
and choose possible solutions. It also can strengthen their resolve to change ineffective practices or 
reinforce decisions and practices that appear to be on the right path. 


This chapter will discuss the remaining elements of the data cycle of collaborative inquiry introduced 
in figure 9 in Chapter Six—analyzing data, organizing data-driven dialogue, drawing conclusions and 
taking action, and monitoring results. 


Step Three: Analyze and Present the Data 


Using postsecondary data for the purpose of helping schools improve their work on postsecondary 
readiness and access does not require advanced statistical calculations. Charts of regression analyses 
are likely to be greeted by most teachers and administrators with blank stares. Rather, simple frequen- 
cies and cross-tabulations are generally what are needed to show the patterns that emerge when the 
data are analyzed. The analyses could be guided by findings from the highly detailed analyses carried 


out by researchers, as described in the Introduction to this guide. 


Data themselves are neutral sets of information. Their power rests in their meaning to different audi- 
ences and in the actions emerging from discussions of that meaning. Therefore, there is no one right 
way to present data to practitioners or a broader community. 


Often, data results must be reconfigured before being shared with others because they appear in 
dense, hard-to-decipher charts with confusing labels. Because people absorb information differently, 
multiple ways should be used to present the same data, including simple charts with numbers, graphic 
presentations that highlight important patterns, and verbal descriptions of what the data show. 


For example, when analyzing the way different factors are related to students’ choices of postsecondary edu- 
cation options, a simple table could show the enrollment of graduates in postsecondary institutions with 
different levels of selectivity. This table would then be followed by a series of tables with the same enroll- 
ment results broken out by the graduates’ characteristics, such as gender and race/ethnicity, or by whether 
students had completed advanced courses in mathematics, or by their GPA and ACT or SAT scores, or by 
whether graduates had submitted FAFSA forms to receive financial aid. Each table could be coupled with a 
graph showing the results, as well as a short narrative description of what the graph and table present. 


When urgent desires for improvement exist, data can appear to be handy tools to prod others to ac- 
tion. Resisting that seemingly easy approach is critical. Punitive use of data leads to low morale, creat- 
ing more resistance than action. Alternatively, a positive use of data can leverage increased school and 
community support for improvement. 


® David Ronka et al., “Answering the Questions That Count,” Educational Leadership (December 2008/January 2009), 18. 
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Step Four: Organize Data-Driven Dialogue 


“Our schools celebrate these data but I'm not sure they use them to impact practices to increase postsecond- 
ary access and success.’ —Smaller Learning Communities Grant Project Director, November 2011. 


When presenting any data, consider some basic ground rules for data presentation: 


Who is the immediate audience? 
Why are you sharing the data with them? 
What do you expect them to do with the information? 


Are the datasets clear and easily understood? To what extent is explanation required to inter- 
pret the data? 


Who is best suited to determine what the data say—the presenter or the audience? 


The answers to these questions can guide choices about what data to present and how to do it. It is 
important to think about a cycle of inquiry and where the particular data presentation sits in that 
cycle. Leaders need to make the best decisions they can at a particular moment in time and then move 
forward confidently, all the while knowing that they may need to make adjustments as new data are 
presented and reviewed as part of the cycle of inquiry for continuous improvement. 


Figure 10 shows three recommended phases of collaborative dialogue to engage practitioners in dis- 
cussion about data in ways that can inform changes in practice. 


Figure 10. Three phases of data-driven dialogue 


Phase 1 
Activating & Engaging 
Surfacing experiences, possibilities, and 
expectations 


With what assumptions are we entering? 
What are some predictions we are 
making? 

What are some questions we are asking? 
What are some possibilities for learning 
what this experience presents for us? 


Phase 2 Phase 3 
Exploring & Discovering Organizing & Integrating 
Analyzing the data Generating hypotheses 


What important points seem to What inferences, explanations, and 
“pop out’? conclusions might we draw? 
What are some patterns or (causation) 
trends that are emerging? What additional data sources might 
What seems to be surprising or we explore to verify our 
unexpected? explanations? (confirmation) 
What are some things we have What are some solutions we might 
not yet explored? explore as a result of our 
conclusions? (action) 
What data will we need to collect to 
guide our implementation? 
(calibration) 
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NOTE: Adapted from Bruce Wellman and Laura Lipton, Data Driven Dialogue: Tools for Collaborative Inquiry (Sher- 
man, CT: Miravia, LLC, 2004), cited in Love, Using Data/Getting Results, p. 45. 


The first phase, Activating and Engaging, lays the groundwork for productive discussions about the 
data by asking the participants to build on their experience and knowledge to think about the central 
questions of their inquiry and what they expect to learn. The second phase, Exploring and Discover- 
ing, asks the participants to take one or two elements of data and examine them, trying to see the pat- 
terns or trends. This phase is about observing what appears in the data rather than trying to explain it. 


Data discussions sometime raise new questions about other variables and how they might affect the 
patterns in the data. In some cases, these additional questions can be anticipated, and the analyses of 
the data needed to answer these new questions will be easily available in the detail that breaks out the 
data by different categories. Sometimes, new questions will require new data collection, such as stu- 
dent surveys or focus groups, to help examine what might explain the patterns in the data. 


The third phase, Organizing and Integrating, moves participants toward generating hypotheses and 
advancing possible explanations for the patterns in the data, helping them to zero in on the underly- 
ing factors that need to become their focus for action. In this phase, participants can discuss what the 
patterns in the data suggest might work to change those they have observed, and begin to stake out 
benchmarks to measure in the future to see whether the action they take has the desired effects. 


Step Five: Draw Conclusions and Take Action 


The next step in the cycle is to draw conclusions about the factors that appear to be most important 
to address in changes in practice. It is important to remember to maintain the focus on the factors 
that can be addressed by the school and district, rather than those external to it. In addition, because 
schools are complex systems, discussions about next steps should place the actions being contem- 
plated in the context of the larger school system, and identify what resources will be needed to move 
forward. 


Step Six: Monitoring Results 


To complete the cycle of inquiry, the next step is to determine a set of concrete and measurable bench- 
marks to monitor progress toward improving postsecondary readiness, access, and success. For ex- 
ample, if the central question of the inquiry has been disparities across groups in the numbers of 
students taking advanced mathematics courses, the progress indicators could include improved grades 
achieved in lower-level mathematics courses (reflecting a stronger foundation for taking advanced 
classes), the number and proportion of students from different groups enrolling in advanced math- 
ematics course work, and their average grades in these classes. 


The following case study shows how this process played out when five high schools in Maine banded 
together to improve student outcomes. 


Maine Smaller Learning Communities Consortium: Data- 
Driven Planning 


In 2006, a consortium of five comprehensive high schools in Maine were awarded a 
Smaller Learning Communities Program (SLCP) grant from the U.S. Department 
of Education to convert their schools into small personalized learning environments. 
In an effort to monitor the progress of the SLCs and gauge their effectiveness on stu- 
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dents, Bonny Eagle, Lewiston, Noble, Oxford Hills, and South Portland high schools 
sought out the University of Massachusetts Donahue Institute (UMDI) to serve as a 
third-party evaluator. 


To carry out the evaluation, UMDI generated a longitudinal student database that 
compares student data by subgroups (e.g., gender, ethnicity, grade, SLC, and course 
grades) and analyzes trends over time. The database also incorporates qualitative data 
such as teacher and student surveys, a longitudinal study done over the life of the 
grant, and annual site visits. The next step was to incorporate NSC data to identify 
postsecondary trends. 


First Step: Central Questions for the Program Evaluation 


UMDI conducted this work over a two-year period, from February 2010 to June 
2011. UMDI’s goal for the first year was to answer the following questions: 


e What is the profile of students who attend postsecondary education? How 
does this compare with students who do not attend further education or pursue 
other options (if available)? 


What happens to high school graduates (by all subgroups including gender, 
race, income status, special education status, etc.) after matriculating in higher 
education? Do they successfully continue? 


Collecting the data: The student data needed to answer these questions came from 
two sources: individual student records from the five high schools and the NSC. In- 
dividual student records were collected from the high schools for the 2006-2007 
through the 2010-2011 school years, including demographic characteristics, grades, 
attendance, academic performance, and postsecondary plans. The NSC consisted 
of data for the 2007, 2008, 2009, and 2010 high school graduates, including col- 
lege attendance, dates of enrollment, names of institutions (two year and four year), 
graduation date, and field of study. Of all the 2007-2010 graduates, 50-71 percent 
of students were found to have attended some form of postsecondary institution, de- 


pending on which of the five schools they attended. 


UMDI’s findings varied slightly across the schools but identified common trends. 
Most of the findings were not surprising: Special Education students, students who 
receive free or reduced-price lunch, and students with poor attendance or violence- or 
alcohol-related suspensions or expulsions or course failures, were less likely to attend 
a postsecondary institution. Performing well on the SAT examinations, mathematics, 
or English courses or taking at least one AP-level course increased a student’s likeli- 
hood of postsecondary matriculation. 


The importance of core course failures. Because of the wide scope of data collected, 
UMDI was able to look at postsecondary persistence as well as access. Students who 
failed a core course were less likely to enroll in college. Among students who failed 
one or more English Language Arts (ELA) courses, only 13 percent enrolled in col- 
lege, versus 59 percent for those students who passed all of their ELA courses. For 
students who failed at least one course in any of the core subjects, 28 percent enrolled 
in college, while 63 percent of students who passed all of their core subjects enrolled. 
Failing a core course also had an effect on persistence in postsecondary education. 
Overall, 53 percent of students who failed one or more core courses persisted in col- 
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lege, while 84 percent of those who passed all of their courses persisted. 
Second Step: Data Retreats 


Once the data have been collected and analyzed, what do you do with it to lead to 
change? Members of the Maine consortium asked themselves that exact question. 
The answer was to engage in a series of data retreats to learn to take part in mean- 
ingful discussions about the data that can then inform an action plan. Each school 
created a four-person team made up of the SLC coordinator, data coordinator, an 
administrator, and a guidance counselor. These teams met three times a year for a 
full-day data retreat. 


The data retreats used protocols from the book The Data Coach’s Guide to Improving 
Learning for All Students.” The central purpose of the retreats was for each team to 
learn how to analyze and discuss data in a way that is meaningful and could inform 
sustainable action. The teams would then take back their new knowledge and skills to 
their respective high schools to share with the rest of the staff. 


The focus of The Data Coach’ Guide is three-pronged: 


e Tocontribute to “dramatic and permanent school improvement” 
To support the ethical determination that school will be good for every student 
e To provide the technical process by which to accomplish purposes one and two 


The retreats focused on two questions: “What are the commonalities among students 
who enroll in postsecondary institutions?” and “What can we do to give all students 
access to these common characteristics?” 


The teams examined the data to discuss what was behind the UMDI evaluation find- 
ings. They began to see that, once past the usual red flags (i.e., attendance, core course 
failure, and behavior), the factor that appeared to be most associated with postsec- 
ondary persistence (being enrolled in a postsecondary institution in the fall of the 
second year following graduation) was taking and being successful in at least one Ad- 
vanced Placement (AP) course. 


Now that a postsecondary student profile had been uncovered, the teams needed to 
put it into action. The action plans varied by school, but all had a common theme: 
to increase the number of students taking AP-level courses. Some of the high schools 
eliminated the “gatekeepers” for their AP courses, such as prerequisite honors cours- 
es, teacher recommendations, or GPA requirements. Another high school partnered 
with the local community college to create a satellite site in the school where high 
school students could take college evening classes for credit. 


These efforts have paid off, as table 10 shows. National Student Clearinghouse data 
indicates that four of the five schools now are sending more students on to postsec- 
ondary education. 


Nancy Love et al., The Data Coach’s Guide to Improving learning for All Students. (Thousand Oaks, CA: Corwin Press, 
2008). 
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Table 10: Percent of graduates enrolled in postsecondary education 


VA0)0}e) VA0)0\6) 2007 2008 yA0}0}°) 2010 gain 


S. Portland 


Source: NSC Reports, Spring 2011. 


The data analysis in this example, as well as the others cited in the earlier profiles, does not depend 
on sophisticated statistical analyses. But with a collaborative review of fairly simple data, sometimes 
with external assistance, teachers and counselors have made invisible factors that were holding their 
students back visible and made targeted action possible. In doing so, they have been able change the 
numbers and to alter the lives of the young people they serve. 
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Closing Remarks 


The advent of longitudinal data systems has greatly increased the capacity of researchers, policy mak- 
ers, and practitioners to learn about students’ trajectories over time and institutions. As this guide is 
being written, the state longitudinal data systems (SLDS) are still evolving, as is the research base on 
what factors matter most in preparing today’s young people to be tomorrow’s successful adults. 


But even as the systems improve and our knowledge grows, the challenge remains in how to translate 
the advances in research and improvements in data access to changes at the classroom and individual 
student level. The approaches described in this guide to working with data with the goal of improving 
the postsecondary chances of students are solid and time-tested ways that practitioners can turn data 
into action. 
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Footnotes 


1 


The asset analysis was conducted using a tool developed by the CPSP technical support partner, FHI 360; it can be found 
at http://www.fhi360.org/resource/navigating-college-readiness-through-asset-mapping-fact-sheet. This tool is based on 
David Conley’s research on college readiness and organized into four categories: academic behavior, cognitive skills, con- 
textual skills and content knowledge. It also draws from Educators for Social Responsibility’s work on increasing college 
access, particularly the development of a continuum of support starting in the ninth grade, as discussed in “A Suggested 
Outline for Postsecondary Preparation,’ from Increasing College Access through School-Based Models of Postsecondary 
Preparation, Planning, and Support. (Cambridge, MA: ESR, January 2009); as well as the Pathways to College Network 
(PCN) College Readiness for All Toolbox, accessed December 10, 2009, http://toolbox.pathwaystocollege.net/. 


FACTS.org is Florida’s Academic Counseling and Tracking for Students. Sponsored by the Department of Education 
and the Florida Center for Advising & Academic Support, this free online advising website helps students plan and 
track their education progress from middle school through college. 


» 


Laura Isensee, “Getting a Jump on College (at a Discount)? Miami Herald, posted Feb. 22, 2012. 
Chicago Public Schools, “CPS students post record FAFSA completion rate,’ press release, February 28, 2011. 


Jay Sherwin, “Make Me a Match: Helping Low-Income and First-Generation Students Make Good College Choices” 
(MDRC Policy Brief, April 2012). 


These include scoring 65+ on the Algebra I, Math B, Chemistry or Physics Regents exam, scoring 3+ on any Advanced 
Placement Exam, scoring 4+ on any International Baccalaureate exam, or earning a C or higher in a course for college 
credit. 


Students can exempt from remedial coursework by reaching score thresholds on the NY State Regents examinations, 
the SAT, and/or course standards accepted by CUNY. 


The A-G requirements include the sequence of high school courses needed to satisfy minimum eligibility requirements 
for the University of California and California State University systems. 


Definition downloaded from http://www.calstate.edu/eap/. This website includes public access to school and district 
test results, private student access to their test results and an analysis of their areas of weakness, as well as support for 
both students and teachers. 


' Description downloaded from http://www.calstate.edu/eap/documents/eap_program_description.pdf. 


1! Nancy Love et al., The Data Coach’s Guide to Improving learning for All Students. (Thousand Oaks, CA: Corwin 


Press, 2008). 
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Appendix A: 
Sample High School Feedback Reports 
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Kentucky 
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Kentucky High School Feedback Report GE High School 
College Going, Class of 2009-2010 Jefferson County Public Schools 


The Kentucky High School Feedback Report is produced by the Kentucky P-20 Data Collaborative, a joint effort from the Kentucky Department 
of Education, Council on Postsecondary Education, Education Professional Standards Board, the Kentucky Education and Workforce 
Development Cabinet, and the Kentucky Higher Education Assistance Authority. This report provides the most complete and accurate data 
about college going that exists for this school. It follows students who graduated from high school in 2009-2010 and were enrolled in a college 
or university during the 2010-2011 academic year. 


| A. Overall, how do graduates from al feLamexenarece) mexe)iil orcla- Mm Come) dal-le-M aM Gla] ducer qa 
School District Kentucky : Sie ; 
Quick Statistics About This 
1. Number of high school graduates 211 5,948 43,711 School 
2. Average high school GPA 2.58 2.65 2.84 
3. Percent of high school graduates in this 80.1% 44.0% 42.6% High School Graduation 79.3% 
class who qualify for free and reduced lunch Rate (KYAFGR) 
4. Average Junior ACT scores for these high 
school graduates by subject Overall College-Going 73.0% 
English 14.8 17.3 17 Bote 
Mathematics 15.7 18.2 18.5 
Reading 16.0 18.5 18.7 *% of Graduates Ready 8.0% 
Science 16.2 18.5 18.8 for college-level math 
Composite 15.8 18.3 18.6 
* 
5. Average Kentucky Education Excellence $727 $1,000 $1,024 HO GUO GNENES Ready, 21 
Scholarship (KEES) Awards earned by vole soe Sila) ely 
these high school graduates 
6. Overall college-going rate for these high 73.0% 62.4% 61.4% 
school graduates * % of Graduates Ready 18.6% 
7. Percent of high school graduates who go 65.4% 55.3% 56.7% for college-level reading 
to an in-state college or university ; : ; 
*Math, English, and reading readiness 
8. Percent of high school graduates who go 8.1% 7.3% 4.8% calculations are based on the average 
to an out-of-state college or university Junior ACT scores for these graduates 


using CPE’s systemwide benchmarks. 


College Going Rates for this Year's Graduating Class 


Percentage 


Geode 


School District 


Note: This high school feedback report was produced using the Kentucky P-20 Data System and provides information at a deeper level and in a different format 
than previous reports. If you intend to compare data from this report to others, please review the technical notes for each to ensure that the data are 
comparable. For more information, please visit http://KentuckyP20.ky.gov__or email P20@ky.gov. 


Fs : 
kentucky P20 7" 


Kentucky High School Feedback Reports BE High School 
College Going, Class of 2009-2010 Jefferson County Public Schools 


B. What types of colleges and universities did graduates from al telamexevelele) mmr lacclale ig 


School District Kentucky 
1. College enrollment of 2009-2010 graduates entering an in-state or independent college (*) 132 3,047 24,066 
2. Type of college or university attended 
a. Four-year public university 67.4% 56.0% 48.2% 
b. Two-year public community or technical college (KCTCS) 31.8% 37.1% 42.0% 
c. Independent 9.1% 10.9% 13.9% 
3. Started college full-time 85.6% 86.8% 92.0% 
4. Type of degree or credential sought 
a. Bachelor's degree 50.8% 54.5% 49.4% 
b. Associate's degree 25.8% 17.1% 23.6% 
c. Certificate or Diploma 6.8% 3.6% 4.0% 
d. Undeclared or no degree 16.7% 24.8% 22.9% 


(*) The P-20 Data Collaborative is able to produce more detailed reporting about graduates who attend in-state public and independent institutions 
than for graduates who attend other in-state private or proprietary and out-of-state institutions. 


College Going Rates for Graduates by Demographic Characteristics 


African American Free Lunch 


Special Ed 
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Note: This high school feedback report was produced using the Kentucky P-20 Data System and provides information at a deeper level and in a different format 
than previous reports. If you intend to compare data from this report to others, please review the technical notes for each to ensure that the data are 
comparable. For more information, please visit http://KentuckyP20.ky.gov__or email P20@ky.gov. 


kent UNBRIDLED uckiy 5 orative 


Kentucky High School Feedback Reports GE High School 
College Going, Class of 2009-2010 Jefferson County Public Schools 


Type of Postsecondary Institution Attended by these High School Graduates 


MH 4-Yr Public \ 2-Yr Public Hi Independent i in-state H Out-of-state H Did not attend 


Proprietary 


Average Junior Year ACT Composite Scores by Enrolled Institution 


08-09 Junior ACT Average 


School District State 
@ 4-Yr Public © 2-Yr Public mw 'ndependent @ In-state Proprietary Hi Out-of-state 


Note: This high school feedback report was produced using the Kentucky P-20 Data System and provides information at a deeper level and in a different format 
than previous reports. If you intend to compare data from this report to others, please review the technical notes for each to ensure that the data are 
comparable. For more information, please visit http://KentuckyP20.ky.gov__or email P20@ky.gov. 
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Kentucky High School Feedback Reports BE High School 
College Going, Class of 2009-2010 Jefferson County Public Schools 


Comm 'JalColamexe) | (-1e(-t-mr-lale MULAN AV(cle-sid(ct-mmel(eMaat-mele-ColUlclcct-Micelin| al felamexevalele) mr-ltclale ig 


Count by College or University Attended 


w 
fy 


Jefferson Community & Technical Colleg 


N 
wo 


University of Louisville 


N 
oO 


Western Kentucky University 


BR 
N 


University of Kentucky 


BR 
N 


Eastern Kentucky University 


BR 
oO 


Spalding University 

Bluegrass Community & Technical Colle 
Indiana University Southeast 

Kentucky State University 

Northern Kentucky University 

Morehead State University 

Spencerian College 

Sullivan University 

Tennessee State University 

Vanderbilt University 

Brown Mackie College Louisville 

Clark Atlanta University 

College Of Wooster 

Community College Of Baltimore County 
Daymar College 

Feather River Community College 
Howard University 

Oakwood University 

Georgetown College 


Murray State University 


PPP BP BP BP BB BB NY NYN NYDN NY Ww Ww DN & 


Campbellsville University 


Note: This high school feedback report was produced using the Kentucky P-20 Data System and provides information at a deeper level and in a different format 
than previous reports. If you intend to compare data from this report to others, please review the technical notes for each to ensure that the data are 
comparable. For more information, please visit http://KentuckyP20.ky.gov__or email P20@ky.gov. 
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Indiana College Readiness Report 


2010 Graduates 


BE High School 


rhs. feecees PNac-aleliaremlacelt-lat-W eae) e)icem exe) | (-10 (=<) 


ME High 


School Statewide 
Count of 2010 graduates 456 69,054 


Unduplicated count of 2010 graduates attending Indiana 
public colleges 232 32,493 


Pam lacclarellale MIN elele)iemexe) (cref-c-M liam aaY(-x- Ug 


+ Graduates Needing Remediation 


MS High 
School Statewide 
Count of 2010 graduates enrolled 232 32,493 
Count needing remediation (Math, E/LA, or Math and E/LA) 45 10,195 


‘aa Remediation by Subject * 


BE High School Statewide 
Subject Count % Count % 
Math Only 24 10% 5,692 18% 
English/Language Arts Only 5 2% 1,051 3% 
Both Math and ngs 0a ade Arts 7% 3,471 11% 


Remediation by Diploma Type * 


BE High School 
Diploma Type 
Subject Honors Core 40 General Unknown/Oth 
Math Only 0 16 5 3 
English/Language Arts Only 0 1 2 2 
Both Math and English/Language Arts 0 4 6 6 


No Remediation 


Statewide 
Diploma Type 
Subject Honors Core 40 General Unknown/Oth 
Math Only 816 3,232 896 758 
English/Language Arts Only 50 491 326 185 
Both Math and English/Language Arts 54 1,497 1,164 758 
No Remediation 12,038 8,236 1,080 1,351 


Total % Needing Remediation 


$3 
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Indiana College Readiness Report 


2010 Graduates 


BE High School 


Enrollment by Degree Type * 


HE High School Statewide 
Degree Type Count % Count % 
Bachelor's degrees 161 69% 21,810 67% 
Associate degrees 64 28% 10,636 33% 
Awards at least 1 but less than 2 academic years 6 3% 247 1% 
Awards of less than 1 academic year 1 0% 167 1% 
Unclassified undergraduate 0 0% 52 0% 


Enrollment by Status * 


HB High School Statewide 
Status Count % Count % 
Full-Time 163 70% 21,616 67% 
Part-Time 69 30% 11,187 34% 


Full-Time = 12 or more total credit hours reported for both fall and spring 


Enrollment by Institution Type and Institution * 


HE High School Statewide 
Institution Type Count % Count % 
2-year Indiana Public Institution 57 25% 10,393 32% 
Ivy Tech Community College of Indiana 54 23% 8,589 26% 
Vincennes University 8) 1% 1,845 6% 
4-year Indiana Public Institution 175 75% 22,486 69% 
Ball State University 16 7% 3,086 10% 
Indiana State University 3 1% 22S 7% 
University Of Southern Indiana 8) 1% 1,816 6% 
Indiana University-Bloomington 29 13% 4,401 14% 
Indiana University-East 0 0% 272 1% 
Indiana University-Kokomo 0 0% 334 1% 
Indiana University-Northwest 0 0% 564 2% 
Indiana University-Purdue University-Indianapolis 3 1% 2,734 8% 
Indiana University-South Bend 0 0% 672 2% 
Indiana University-Southeast 0 0% 795 2% 
Indiana University-Purdue University-Fort Wayne 97 42% 1,368 4% 
Purdue University-Calumet Campus 0 0% 780 2% 
Purdue University-North Central Campus 0 0% 519 2% 
Purdue University-West Lafayette Campus 25 11% 3,607 11% 
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A Enrollment by Program Type * 


A+ 
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Indiana College Readiness Report 


2010 Graduates 


BE High School 


HB High School Statewide 

Program Type Count % Count % 
Education 28 12% 2,928 9% 
Arts and Humanities 29 13% 5,101 16% 
Social and Behavioral Sciences and Human Services 19 8% 2,693 8% 
Science, Technology, Engineering, and Math (STEM) 56 24% 6,115 19% 
Business and Communication 49 21% 4,719 15% 
Health 25 11% 2,744 8% 
Trades 11 5% 1,941 6% 
Undecided 15 6% 6,852 21% 

BE High 
Diploma Type School Statewide 
Honors 3.3 3.1 
Core 40 2.4 2.2 
General 2.2 1.9 
Unknown/Other 1.7 2.1 


p x Average Credit Hours Earned by Institution Type and Status 


HB High 


School Statewide 


2-year Institutions 


Full-Time ZS eee 
Part-Time ue ee 
4-year Institutions 

Full-Time zis ali 
Part-Time 2A 11.9 


Full-Time = 12 or more total credit hours reported for both fall and spring 


* Data in this report represent the entire 2010-2011 fiscal year. Some students may have been enrolled at more 
than one institution during FY2010-2011. Those students may appear more than once in some tables. As such, 
counts in each table may not equal total count of students attending a public postsecondary institution and 
percentages may not equal 100%. 


*** Cells with fewer than 10 students represented are suppressed for privacy reasons 


Data sources: Total 2010 graduate counts: Indiana Department of Education GR (graduates) report; 
All other data: Commission for Higher Education SIS data submissions (FY2011) 
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